Reference selection for video interpolation or extrapolation

ABSTRACT

This disclosure describes selection of reference video units to be used for interpolation or extrapolation of a video unit, such as a video frame. A decoder may apply a quality-focused mode to select a reference frame based on quality criteria. The quality criteria may indicate a level of quality likely to be produced by a reference frame. If no reference frames satisfy the quality criteria, interpolation or extrapolation may be disabled. Display of an interpolated or extrapolated frame may be selectively enabled based on a quality analysis. A decoder may apply a resource-focused frame interpolation mode to enable or disable frame interpolation or extrapolation for some frames based on power and quality considerations. In one mode, frame interpolation may be disabled to conserve power when reference frames are not likely to produce satisfactory quality. In another mode, the threshold may be adjustable as a function of power saving requirements.

This application claims the benefit of U.S. provisional application No.61/012,703, filed Dec. 10, 2007, the entire content of which isincorporated herein by reference.

TECHNICAL FIELD

The disclosure relates to digital video coding and, more particularly,techniques for video frame interpolation or extrapolation.

BACKGROUND

A number of video encoding techniques have been developed for encodingdigital video sequences. The Moving Picture Experts Group (MPEG), forexample, has developed several techniques including MPEG-1, MPEG-2 andMPEG-4. Other examples include the International Telecommunication Union(ITU)-T H.263 standard, and the ITU-T H.264 standard and itscounterpart, ISO/IEC MPEG-4, Part 10, i.e., Advanced Video Coding (AVC).These video encoding standards support efficient transmission of videosequences by encoding data in a compressed manner. Compression reducesthe overall amount of data that needs to be transmitted.

Video compression may involve spatial and/or temporal prediction toreduce redundancy inherent in video sequences. Intra-coding uses spatialprediction to reduce spatial redundancy between video blocks within thesame video frame. Inter-coding uses temporal prediction to reducetemporal redundancy between video blocks in successive video frames. Forinter-coding, a video encoder performs motion estimation to generatemotion vectors indicating displacement of video blocks relative tocorresponding prediction video blocks in one or more reference frames.The video encoder performs motion compensation to generate a predictionvideo block from the reference frame, and forms a residual video blockby subtracting the prediction video block from the original video blockbeing coded.

To meet low bandwidth requirements, some video applications may encodevideo at a reduced frame rate and/or skip encoding of some frames.Unfortunately, low frame rate video can produce temporal artifacts inthe form of motion jerkiness. Frame interpolation or extrapolation maybe employed at the decoder side to approximate the content of framesthat are skipped by the encoder, or frames that are beyond the basicframe rate produced by the encoder. Frame interpolation or extrapolationmay generally be referred to as frame substitution. Frame substitutionmay be used, in effect, to upconvert an actual frame rate to provide aperception of smoother motion. Frame substitution may be used to supporta process often referred to as frame rate up conversion (FRUC). AlthoughFRUC may enhance temporal quality by substituting frames, e.g., usinginterpolation or extrapolation, substitution of some frames mayintroduce undesirable spatial artifacts that undermine visual quality.

SUMMARY

This disclosure is directed to techniques for selecting one or morereference video units to be used for substitution of a video unit, e.g.,by interpolation or extrapolation. The video units may be video frames,slices, blocks, or other units. A video coder may apply aquality-focused video mode to select a reference video unit based onanalysis of one or more quality criteria. The quality criteria mayindicate, for example, a level of interpolation or extrapolation qualitylikely to be produced by a selected reference video unit. Qualitycriteria may include spatial and/or temporal visual quality. If none ofthe reference video units satisfies applicable quality criteria,substitution may be disabled for a particular video unit to be added.

A video coder may apply a resource-focused video mode to selectivelyenable or disable substitution for some video units based on a motionlevel of the video sequence. If one or more reference video units aregenerally static, the video decoder or another device may disablesubstitution, thereby conserving resources such as power, computingand/or memory resources. The motion level may be compared to athreshold, which may be fixed or adjustable as a function of a level ofavailable resources. If the reference video units include substantialmotion, the video decoder or another device may enable substitution,e.g., using quality criteria for selection of reference video units.

For delay-sensitive video applications, such as video telephony, a videocoder or another device may be configured to select video referencevideo units to reduce processing and presentation delay. For example,when selecting future reference video units, a video coder may beconfigured to select reference video units based on distance from avideo unit to be added. A video coder also may be configured to analyzeone or more quality characteristics associated with a video unit thathas been interpolated or extrapolated, and selectively enable or disabledisplay of the video unit based on the analysis, thereby conservingresources needed to display the additional video unit in some cases.

In one aspect, the disclosure provides a method comprising analyzing atleast one characteristic of one or more candidate reference video units,and selecting one or more of the candidate reference video units as areference video unit for interpolation or extrapolation of an additionalvideo unit based at least in part on the analysis.

In another aspect, the disclosure provides a device comprising ananalysis unit that analyzes at least one characteristic of one or morecandidate reference video units, and a selection unit that selects oneor more of the candidate reference video units as a reference video unitfor interpolation or extrapolation of an additional video unit based atleast in part on the analysis.

In an additional aspect, the disclosure provides a method comprisinganalyzing a motion level of one or more candidate reference video unitsfor interpolation or extrapolation of an additional video unit,determining a resource level for interpolation or extrapolation of theadditional video unit, and selectively disabling interpolation orextrapolation of the additional video unit based on the motion level andthe resource level.

In a further aspect, the disclosure provides device comprising a motionanalyzer configured to analyze a motion level of one or more candidatereference video units for interpolation or extrapolation of anadditional video unit, a resource monitor configured to determine aresource level for interpolation or extrapolation of the additionalvideo unit, and a selection unit that selectively disables interpolationor extrapolation of the additional video unit based on the motion leveland the resource level.

In another aspect, the disclosure provides a video decoding devicecomprising an analysis unit that analyzes one or more characteristicsassociated with an interpolated or extrapolated video unit produced by aframe rate upconversion process, and a control unit that selectivelydisables presentation of the interpolated or extrapolated video unit ona display based on the analysis.

In an additional aspect, the disclosure provides a method comprisinganalyzing one or more characteristics associated with an interpolated orextrapolated video unit produced by a frame rate upconversion process,and selectively disabling presentation of the interpolated orextrapolated video unit on a display based on the analysis.

The techniques described in this disclosure may be implemented inhardware, software, firmware, or a combination thereof. If implementedin software, the software may be executed by one or more processors. Thesoftware may be initially stored in a computer readable medium andloaded by a processor for execution. Accordingly, this disclosurecontemplates computer-readable media comprising instructions to causeone or more processors to perform techniques as described in thisdisclosure.

For example, in some aspects, the disclosure provides acomputer-readable medium comprising instructions to cause one or moreprocessors to analyze at least one characteristic of one or morecandidate reference video units, and select one or more of the candidatereference video units as a reference video unit for interpolation orextrapolation of an additional video unit based at least in part on theanalysis.

In other aspects, the disclosure provides a computer-readable mediumcomprising instructions to cause one or more processors to analyze amotion level of one or more candidate reference video units forinterpolation or extrapolation of an additional video unit, determine aresource level for interpolation or extrapolation of the additionalvideo unit, and selectively disable interpolation or extrapolation ofthe additional video unit based on the motion level and the resourcelevel.

In another aspect, the disclosure provides a computer-readable mediumcomprising instructions to cause one or more processors to analyze oneor more characteristics associated with an interpolated or extrapolatedvideo unit produced by a frame rate upconversion process, andselectively enable and disable presentation of the interpolated orextrapolated video unit on a display based on the analysis.

The details of one or more aspects of the disclosed techniques are setforth in the accompanying drawings and the description below. Otherfeatures, objects, and advantages will be apparent from the descriptionand drawings, and from the claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating a video encoding and decodingsystem configured to select reference video units for use in video unitsubstitution.

FIG. 2A is a diagram illustrating an example of a technique forinterpolation of a video unit in a video decoder.

FIG. 2B is a diagram illustrating interpolation of a video unit usingselected reference video units.

FIG. 2C is a diagram illustrating an example of a technique forextrapolation of a video unit in a video decoder.

FIG. 2D is a diagram illustrating extrapolation of a video unit usingselected reference video units.

FIG. 3 is a block diagram illustrating an example of a video decoderconfigured to select reference frames for use in frame substitution.

FIG. 4 is a block diagram illustrating another example of a videodecoder configured to select reference frames for use in framesubstitution.

FIG. 5 is a block diagram illustrating an analysis unit that may be usedwith a video decoder as shown in FIG. 3 or FIG. 4.

FIG. 6 is a flow diagram illustrating an example technique in which avideo decoder selects reference video units for video unit substitution.

FIG. 7 is a flow diagram illustrating an example technique for referencevideo unit selection in more detail.

FIG. 8 is a flow diagram illustrating an example technique for qualityanalysis of reference video units to support reference video unitselection for video unit substitution.

FIG. 9 is a flow diagram illustrating an example technique forgenerating quality scores for reference video units to support referencevideo unit selection for video unit substitution.

FIG. 10 is a flow diagram illustrating an example technique forselective substitution based on motion analysis in a resource-focusedmode.

FIG. 11 is a flow diagram illustrating another example technique forselective substitution based on motion analysis in a resource-focusedmode.

FIG. 12 is a block diagram illustrating an example of a video decoderconfigured to selectively enable or disable display of substituteframes.

FIG. 13 is a flow diagram illustrating an example technique forselective display of substitute frames based on quality analysis.

FIG. 14A is a block diagram illustrating an analysis unit that may beused with a video decoder as shown in FIG. 12.

FIG. 14B is a block diagram illustrating another analysis unit that maybe used with a video decoder as shown in FIG. 12.

FIG. 15 is a flow diagram illustrating an example technique forgenerating quality scores to support selective display of substituteframes.

FIG. 16 is a flow diagram illustrating an example technique for qualityanalysis of reference video units to support reference video unitselection for video unit substitution when the video units support adelay-sensitive video application.

DETAILED DESCRIPTION

FIG. 1 is a block diagram illustrating a video encoding and decodingsystem 10 configured to select reference video units for use in videounit substitution. In various aspects, the substituted video units andselected reference video units may be, for example, video frames, videoslices, or video blocks. As shown in FIG. 1, system 10 may include avideo encoder 12 and a video decoder 14, each of which may be generallyreferred to as a video coder. In the example of FIG. 1, video encoder 12encodes input video frames 16 to produce encoded video frames 18. Videoencoder 12 may transmit encoded video frames 18 to video decoder 14 viaa communication channel 19.

Although the techniques described in this disclosure may be applicableto a variety of video units, such as frames, slices, blocks orsub-blocks, this disclosure will generally describe application of thetechniques to video frames for purposes of illustration, but withoutlimitation of the aspects of such techniques as broadly described inthis disclosure.

To reduce the amount of data that must be transmitted between encoder 12and decoder 14, and thereby adhere to reduced bandwidth requirements forchannel 19, video encoder 12 may operate at a basic video unit codingrate that is less than a source video unit coding rate. For example,video encoder 12 may operate at a reduced video frame rate, such as,e.g., 15, 30, 60 frames per second (fps).

Alternatively, or additionally, in some cases, video encoder 12 mayoperate at a given video frame rate, but optionally include orselectively activate a skip unit 20 that causes encoder 12 to skipencoding of some video units. For example, video unit skip unit 20 maybe configured to cause encoder 12 to skip the encoding of some frames,thereby reducing the effective frame rate of video encoder 12, e.g.,relative to a source video frame rate. In FIG. 1, a skipped video frameis illustrated by the shaded frame in encoded frames 18.

In the case of skipped frames or a coding rate that can be upconverted,it may be desirable to substitute additional video units at the decoderside, e.g., by interpolation or extrapolation, to convert the actualvideo frame rate to an increased video frame rate. Such a process issometimes referred to as frame substitution to support frame rate upconversion (FRUC). In effect, decoder 14 may increase the actual framerate produced by video encoder 12 to an up-converted frame rate.

As an example, if the actual frame rate produced by encoder 12, with orwithout frame skipping, is 30 fps, then decoder 14 may be configured tosubstitute additional frames, e.g., by interpolation or extrapolation,to increase the effective frame rate from 30 fps to 60 fps or 120 fps.In effect, the additional frames are substituted for frames that havebeen skipped or frames that could have been included if the basic framecoding rate of video encoder 12 was greater. As described above, theframe rate produced by video encoder 12 may be less than desired due toa basic frame rate that is less than a source video rate and/or theoptional skipping of some frames, e.g., by optional skip unit 20.

By skipping some frames or coding fewer frames per second, as describedabove, video encoder 12 may encode video at a reduced frame rate.However, low frame rate video can produce temporal artifacts in the formof motion jerkiness. Frame substitution may be employed by decoder 14 toapproximate the content of skipped frames or other excluded frames and,in effect, upconvert the actual frame rate to provide a perception ofsmoother motion. For example, video decoder 14 may include a frame rateup conversion (FRUC) unit 22 that interpolates or extrapolates at leastsome additional video frames to increase the effective frame rate of thedecoded video.

Again, although FRUC unit 22 is described with respect to up-conversionof the effective coding rate for frames, the techniques described inthis disclosure may be applied to other video units, such as slices,blocks, or sub-blocks. Video decoder 14 may decode received frames 24and approximate additional video frames via FRUC unit 22 to produceoutput video frames 26. The decoded output video frames 26 may be usedto drive a display device. In FIG. 1, the example of a skipped frame isillustrated by the shaded video frame in received video frames 24.

In the example of FIG. 1, FRUC unit 22 is shown within video decoder 14.In other implementations, FRUC unit 22 may form part of a videopost-processing module. A video post-processing module may process theoutput of video decoder 14, and may perform a variety of processingoperations such as smoothing, sharpening, brightness control, and/orcontrast enhancement, as well as a FRUC operation. As anotheralternative, FRUC unit 22 may form part of a video display processor ormobile display processor (MDP) device, e.g., for a mobile multimediadevice. Accordingly, implementation of FRUC unit 22 within video decoder14 is depicted in FIG. 1 and other figures for purposes of illustration,and should not be considered limiting of the techniques broadlydescribed in this disclosure.

Motion-compensated (MC) video frame interpolation (VFI) is an example ofa technique used to enhance the temporal perceptual quality of video inapplications such as FRUC at the decoder side. Other interpolationtechniques, as well as extrapolation techniques, may be applied toapproximate additional frames to support a FRUC process. Although FRUCtechniques may enhance temporal quality by approximating skipped frames,or generating additional frames beyond the basic frame rate of the videoencoder 12, interpolation or extrapolation of some frames may introduceundesirable spatial artifacts that undermine visual quality.

For example, the visual quality of a substituted video frame may not beguaranteed and may be highly dependent on the particular reference frameor frames that are used to perform the interpolation or extrapolation.In addition, VFI methods may be very complex and consume a great deal ofpower and other resources, which may discourage the use of VFI for videoapplications in some devices, such as mobile devices with limited power,computing and/or memory resources. Other frame substitution techniquesmay present similar quality and resource issues.

FRUC unit 22 may be configured to analyze at least one characteristicassociated with one or more reference video frames received by videodecoder 14, and select one or more of the reference video frames for usein substitution of a video frame by the video decoder based on theanalysis. The reference video frames may be selected from receivedframes 24 that reside temporally prior to or subsequent to a frame to besubstituted. In other words, FRUC unit 22 may select one or moreprevious or future frames 24 for use in approximating an additionalframe to be substituted.

A previous video frame may include a frame that immediately precedes theframe to be substituted, or one or more prior frames in proximity to theframe to be substituted. A future frame may include a frame thatimmediately follows the frame to be substituted, or one or more framesin proximity to the frame to be substituted. In the case of substitutionby interpolation, one or more previous frames and one or more futureframes may be used to interpolate an additional, intermediate frame. Inthe case of substitution by extrapolation, one or more previous framesor one or more future frames may be used for extrapolation of anadditional, previous or future frame.

In some aspects, FRUC unit 22 may analyze the quality of the referencevideo frames to select one or more reference frames for use insubstitution of an additional video frame. In this manner, FRUC unit 22determines which frames to use as reference frames for video framesubstitution, e.g., by interpolation or extrapolation. In this case,FRUC unit 22 may select reference video frames to enhancespatio-temporal video quality of output video frames 26. In otheraspects, FRUC unit 22 may analyze both the quality of the referenceframes and resource constraints of a device in which video decoder 14resides. In this case, FRUC unit 22 may enhance spatio-temporal videoquality of output video frames 26 while balancing interests in reducingpower consumption, conserving computing resources, and/or conservingmemory resources. FRUC unit 22 may enhance interpolated or extrapolatedframe quality as well as temporal quality of the video sequence. Ingeneral, consumption of computing and memory resources may contribute toincreased power consumption, as well as latency in some cases.

In addition, in some aspects, FRUC unit 22 may be configured to selectreference video frames with a bias toward reducing end-to-end processingand/or presentation delays. Such delays may be especially undesirablefrom some real-time or near real-time applications, such as videotelephony, which may be delay-sensitive. For example, when futurereference video frames are used to approximate a substituted frame, FRUCunit 22 may be configured to favor selection of future reference videoframes that are relatively close in time to the frame to beapproximated. Alternatively, FRUC unit 22 may disable video framesubstitution for such delay-sensitive applications.

In a video telephony application, as an illustration, it may lessdesirable to select frame extrapolation based on future frames or, inthe case of video frame interpolation, it may be desirable to selectcloser future frames rather than future frames that are further in thefuture in order to reduce end-to-end delay, and thereby preservetemporal quality for the user. In particular, reliance on futurereference frames that reside further in the future may result in delaysdue to need to wait for such future frames to be decoded. The furtherthe frame resides in the future, the longer the wait may be, which maycause disconcerting delays in the video telephony presentation.

In some aspects, video decoder 14 may provide a quality-focused framesubstitution mode as a first operational mode to select a referenceframe based on analysis of one or more reference frame quality criteria.In addition, video decoder 14 may provide a resource-focused framesubstitution mode as a second operational mode to selectively enable ordisable frame substitution for some frames based on a combination ofresource and quality considerations. The quality-focused andresource-focused modes, in some aspects, may be referred to asquality-optimized and power-optimized modes. Hence, in some aspects,video decoder 14 may decide which frames to use as reference frames forvideo frame interpolation or extrapolation and also decide which framesto interpolate or extrapolate in order to save power and to enhanceinterpolated frame quality as well as temporal quality of video.Alternatively, or additionally, video decoder 14 may be configured todisable transmission of substitute frames from a video buffer to adisplay, even after interpolation or extrapolation has been performed,based on quality criteria.

In some aspects, this resource-focused mode may be considered apower-optimized mode, as discussed above. For example, video decoder 14may be configured to balance visual quality versus power conservationand/or computing load. In some cases, video decoder 14 may beselectively switchable between a quality-focused mode and aresource-focused mode, e.g., according to resources available to thevideo decoder at the time a FRUC operation for a frame to be substitutedis applied.

In the quality-focused mode and/or resource-focused mode, the qualitycriteria may include, for example, one or more characteristicsindicative of a level of substituted frame quality likely to be producedusing a selected reference frame or frames. In other words, thecharacteristics may be selected as an indication of the likely qualityof a frame that is interpolated or extrapolated using the referenceframe. If none of the reference frames satisfies the quality criteria,video decoder 14 may disable frame substitution for a particular frame.Hence, in the quality-focused mode, video decoder 14 may disable frameinterpolation or extrapolation to conserve power when none of thereference frames is likely to produce interpolation or extrapolationquality that is satisfactory, e.g., above a threshold.

In some aspects, the quality-focused criteria may be applied before orafter a substituted frame is actually generated by decoder 14. Forexample, the quality analysis may be applied after frame interpolationor extrapolation, in which case the substitute frame may be selectivelyapplied to a display device based on the outcome. If the quality of aninterpolated or extrapolated frame does not satisfy a threshold qualitylevel, FRUC unit 22 may discard the interpolated or extrapolated framerather than send it from an output video frame buffer to drive a displaydevice.

In this case, even though interpolation or extrapolation has alreadybeen performed, discarding the frame may still be advantageous if thequality level does not justify the additional resources required todisplay the frame. A substantial amount of power may be expended insending a frame from a video buffer to a display buffer to drive thedisplay. Accordingly, discarding a substitute frame, even afterinterpolation or extrapolation has been performed, may save powerconsumption that otherwise would result from video data traffic betweenthe video buffer and the display.

In the resource-focused mode, video decoder 14 may disable framesubstitution if a motion level of one or more candidate reference framesis less than a threshold. In this case, when a video scene is generallystatic, the difference between an interpolated or extrapolated frame anda repeated frame may be negligible. Accordingly, using frame repetitioninstead of interpolation or extrapolation can conserve resources, suchas power. The motion level threshold may be fixed or adjustable as afunction of the resource level or resource saving requirements of thevideo decoder. In either case, whether the quality-focused orresource-focused mode is activated, if frame substitution is enabled,one or more quality criteria may be used to select reference frames.

The one or more quality criteria used to analyze a candidate referenceframe may be selected as characteristics indicating a quality ofinterpolation or extrapolation likely to be produced by using thecandidate reference frames. For example, the quality criteria mayindicate a likely quality of the interpolated or extrapolated frame ifthe candidate reference frame under consideration is used in theinterpolation or extrapolation of that additional frame. Also, FRUC unit22 may analyze motion vector reliability as a further indication ofreference frame quality. Examples of quality criteria analyzed by FRUCunit 22 may include quantization parameter (QP) values, coded blockpattern (CBP) values and the number of non-zero transform coefficientsassociated with reference video frames. If the CBP value is not equal tozero, the QP values may be coupled with the number of non-zerocoefficients to judge the reliability of motion vectors provided by thereference frames.

FRUC unit also may consider objective quality metrics such as astructural similarity metric (SSIM), blockiness, and/or blurriness todetermine the quality of candidate reference frames for use ininterpolation or extrapolation. In addition, the type of intra mode inaddition to intra mode and motion vector count for a reference frame maybe considered.

FRUC unit 22 may analyze other quality criteria such as evidence of fullor partial video unit loss, which may be generally referred to as videounit loss. For example, FRUC unit 22 may analyze slice or frame lossesfor a reference video frame in conjunction with the lack of availabilityof an error concealment mechanism for the frame. For example, FRUC unit22 may evaluate a level of the error and a quality of an errorconcealment mechanism. Other quality criteria may be used in addition,or as an alternative, to the types of quality criteria described above.

In some cases, FRUC unit 22 may select reference video frames thatsatisfy one or more quality thresholds. In other cases, FRUC unit 22 mayscore and rank quality for multiple reference video frames, and selectone or more reference frames producing the best scores. If two frames(e.g., two previous frames or two future frames) are rankedsubstantially the same, it may be desirable to select the frame that istemporally closer to the skipped frame to be interpolated.

For purposes of illustration, this disclosure generally refers to theselection of reference frames for interpolation or extrapolation ofadditional frames. In some implementations, however, the disclosurecontemplates more generally the selection of reference video units forapproximation of additional video units, other than frames. For example,the techniques described in this disclosure may be adapted to analyzeand select any of a variety of reference video units such as videoframes, video slices, or video blocks such as macroblocks.

When a video unit such as a frame, slice or block is skipped by videoencoder 12, the techniques may be used to identify a correspondingframe, slice or block among various candidate reference frames that canbe used for interpolation or extrapolation. Alternatively, even when aframe, slice or block is not skipped, the techniques may be used toapproximate additional frames to augment the basic frame coding rate ofencoder 12, e.g., for frame rate conversion applications.

Even when it is desirable to approximate an entire frame due to skippingor low frame rate, it may be advantageous to select individual slices orblocks for use in interpolating the slices or blocks of the frame to beapproximated. In this case, slices or blocks from different candidateframes may be selected to interpolate or extrapolate correspondingslices or blocks in the frame to be approximated. For example, qualityanalysis or analyses similar to those described in this disclosure maybe applied on a slice-by-slice or block-by-block basis to selectreference video units for interpolation or extrapolation of additionalvideo units. Accordingly, by focusing on selection of reference framesfor interpolation or extrapolation of skipped frames to be substituted,the disclosure should not be considered limiting of the aspects asbroadly described.

With further reference to FIG. 1, video encoder 12 and video decoder 14may be connected by transmission channel 19. Transmission channel 19 maybe a wired or wireless medium, or a combination of both, capable ofconveying video frames within a bitstream. Channel 19 may supportbi-directional or uni-directional video transmission. System 10 may beconfigured for video telephony, video streaming, video broadcasting, orthe like. Accordingly, reciprocal encoding, decoding, multiplexing (MUX)and demultiplexing (DEMUX) components may be provided on opposite endsof channel 19. In some implementations, encoder 12 and decoder 14 may beprovided within video communication devices such as wireless mobileterminals equipped for video streaming, video broadcast reception,and/or video telephony, such as so-called wireless video phones orcamera phones.

Such wireless communication devices include various components tosupport wireless communication, audio coding, video coding, and userinterface features. For example, a wireless communication device mayinclude one or more processors, audio/video encoders/decoders (CODECs),memory, one or more modems, transmit-receive (TX/RX) circuitry such asamplifiers, frequency converters, filters, and the like. In addition, awireless communication device may include image and audio capturedevices, image and audio output devices, associated drivers, user inputmedia, and the like.

Encoder 12, decoder 14 or both may be incorporated in a wireless orwired communication device as described above. Also, encoder 12, decoder14 or both may be implemented as an integrated circuit device, such asan integrated circuit chip or chipset, which may be incorporated in awireless or wired communication device, or in another type of devicesupporting digital video applications, such as a digital media player, apersonal digital assistant (PDA), a digital television, or the like.

System 10 may support video telephony according to the Session InitiatedProtocol (SIP), ITU-T H.323 standard, ITU-T H.324 standard, or otherstandards. Video encoder 12 may generate encoded video data according toa video compression standard, such as MPEG-2, MPEG-4, ITU-T H.263, ITU-TH.264, or MPEG-4, Part 10. Although not shown in FIG. 1, video encoder12 and video decoder 14 may be integrated with an audio encoder anddecoder, respectively, and include appropriate hardware and softwarecomponents to handle both audio and video portions of a data stream.

Encoder 12 encodes video frames 16, which may include Intra frames (Iframes), predictive frames (P frames), and bi-directional predictiveframes (B frames). I frames are frames that completely encode all videoinformation using spatial coding techniques. The encoded frames maycomprise information describing a series of video blocks that form aframe. The video blocks, sometimes referred to as macroblocks (MBs), mayinclude coding bits that define pixel values, e.g., in luminance (Y),chrominance red (Cr) and chrominance blue (Cb) color channels.

As described above, encoder 12 may implement frame skipping to reducethe frame rate of data transmitted over transmission channel 19. Inparticular, encoder 12 may be configured to intentionally skip selectedframes, e.g., by not coding selected frames or not transmitting selectedcoded frames. Alternatively, encoder 12 may generate frames at a basicframe coding rate, with or without frame skipping, that is less than adesired frame rate. Frame skipping or reduced frame rate coding maypermit encoder 12 to conform to a reduced transmission rate requirementof channel 19.

In the case of frame skipping, frames may be skipped by frame skip unit20 at a fixed rate such that skipping occurs at alternating frames or atevery nth frame. Alternatively, frames may be skipped at a varying rate,e.g., based on intelligent frame skipping criteria. Also, encoder 12 mayencode frames at a given frame rate on a fixed basis, or an adaptivebasis such that frame rate is varied according to considerations such aschannel conditions or other requirements. In either case, the frame ratemay be effectively up-converted by the decoder 14 to produce anincreased frame rate, e.g., from 30 fps to 60 fps or 120 fps.

FRUC unit 22 may be used by decoder 14 to perform frame substitution toreplace frames that are skipped or generate additional frames to augmentthe frame rate. In addition, in some implementations, FRUC unit 22 maybe used for frames that are dropped or lost during transmission. Framesskipped by encoder 12 and frames dropped or lost during transmission maygenerally be referred to as skipped frames in this disclosure. In eachcase, to increase the effective frame rate, and thereby improve temporalquality, FRUC unit 22 of decoder 14 may perform a FRUC process toapproximate at least some of the skipped frames with substitute framesusing video frame interpolation or extrapolation and reference frameselection.

If none of the reference video frames is likely to support framesubstitution with a sufficient quality level, FRUC unit 22 may disableframe substitution and apply frame repetition. When frame substitutionis disabled, decoder 14 may simply repeat a previous or future frame,instead of interpolating or extrapolating a frame between the previousframe and the future frame. In this case, decoder 14 may use a duplicateversion of a previous or future frame in place of a skipped frame or asan additional frame for frame rate conversion.

By using frame repetition, decoder 14 may avoid undesirable spatialartifacts that could be introduced by interpolation or extrapolation.Because frame repetition may reduce the perceived temporal quality ofthe video, frame substitution by interpolation or extrapolation willordinarily be more desirable when sufficient quality can be achieved.However, excessive consumption of power, computing, and/or memoryresources may reduce the overall value of substitution techniques. Aresource-focused mode, as described in this disclosure, may serve tobalance quality versus resource consumption.

In the example of FIG. 1, video encoder 12 receives input frames 16(F_(t−2), F_(t−1), F_(t), F_(t+1), F_(t+2)) of video information. F_(t)represents a frame that is not coded at time t, either due to frameskipping by optional frame skip unit 20 or due to the basic frame rateproduced by encoder 12. Accordingly, it should be noted that framesubstitution, as described in this disclosure, generally refers to theaddition of a frame F_(t)′ that approximates a frame F_(t) that was notprovided in the frames received by decoder 14, either due to frameskipping, channel loss, or the basic frame rate of encoder 12.

If frame skip unit 20 is applicable, frames may be skipped according toa fixed, adjustable or dynamic frame skipping process, as describedabove. F_(t−2) and F_(t−1) represent past frames that are temporallyprior to frame F_(t), and F_(t+1) and F_(t+2) are future frames that aretemporarily after frame F_(t). Reference frames that may be used forinterpolation or extrapolation of frame F_(t) may include numerousframes both before and after frame F_(t). For ease of illustration,however, only two frames before and two frames after F_(t) are shown inFIG. 1.

In general, video encoder 12 encodes input frames 16 as one of the abovedescribed I, P, or B frames to produce encoded frames 18. Again, frameF_(t) resides temporally between previous frames F_(t−2), F_(t−1) andfuture frames F_(t+1), F_(t+2). Video encoder 12 transmits encodedframes 18, which includes encoded frames F_(t−2), F_(t−1), F_(t+1),F_(t+2), but not frame F_(t), to video decoder 14 via transmissionchannel 19. Typically, encoder 12 transmits these frames in apre-defined sequence, such as IBBPBBPBBPBBI, where I, B, and P refer toI frames, B frames, and P frames, respectively.

Encoded frames 18 may be intra-coded or inter-coded, and may be decodedto produce video content present in input frames 16. In addition,encoded frames 18 may serve as reference frames for decoding of otherinter-coded frames in a video sequence, i.e., as a reference for motionestimation and motion compensation of a predicted frame. As is wellknown in the art of predictive coding, an encoded frame may becharacterized by motion vectors that indicate displacement of blocks inthe encoded frame relative to similar, corresponding blocks in adifferent encoded frame, which serves as a reference frame. In addition,the encoded frame may be characterized by residual information thatindicates differences between video blocks in the encoded frame andcorresponding video blocks in the reference frame.

Encoding of input frames 16 and decoding of received frames 24 may relyon reference frames as described above for predictive coding. In thecase of a frame to be substituted, however, a reference frame asdescribed in this disclosure generally refers to a frame that is usedfor interpolation or extrapolation to provide an additional frame at thedecoder side. Hence, it should be noted that a reference frame forinterpolation or extrapolation is different in its usage from areference frame for predictive coding, even though a given frame may beused as a reference frame for both interpolation and predictive codingin some instances. Reference frames for predictive coding are specifiedat the encoder side and used for predictive coding. On the contrary,reference frames for interpolation or extrapolation may be selected atthe decoder side and used for substitution of additional frames, e.g.,by interpolation or extrapolation.

In the example of FIG. 1, frames 16 represent five frames in a videosequence containing many frames, and are used to describe theinterpolation or extrapolation of an additional, i.e., extra, frameF_(t)′ that approximates a frame F_(t) that resides temporally betweenencoded frames 18 (e.g., between previous frames F_(t−2), F_(t−1) andfuture frames F_(t+1), F_(t+2)). In some aspects, multiple frames may beneed to be added, in which case more than one frame residing between twotransmitted frames may require interpolation or extrapolation. For easeof illustration, this disclosure will refer to the example case in whicha single frame F_(t)′ between a previous frame and a future frame isinterpolated using selected reference frames.

Video decoder 14 receives frames 24 via transmission channel 19.Received frames 24 may be substantially identical to the encoded frames18 transmitted by video encoder 12, subject to slice, frame or blocklosses due to characteristics of transmission channel 19. Video decoder14 may apply standard decoding techniques to decode each of the receivedframes 24 (F_(t−2), F_(t−1), F_(t+1), F_(t+2)), e.g., according to oneof the MPEG-1, MPEG-2, MPEG-4, H.263, H.264, or MPEG-4, Part 10standards. In accordance with this disclosure, decoder 14 furtherincludes FRUC unit 22, which it applies to frames 24 in order toselectively approximate the additional frame F_(t)′. With FRUC unit 22,video decoder 14 interpolates or extrapolates a frame to generate frameF_(t) and generates output frames 26, which include the decoded framesF_(t−2), F_(t−1), F_(t+1), F_(t+2) and an approximation F_(t)′ of theframe F_(t) to be added.

FRUC unit 22 receives successive frames in the video sequence. For eachadditional frame to be interpolated, there is at least one previousframe and at least one future frame that can be used as reference framesfor interpolation. The frame to be interpolated resides temporallybetween multiple previous frames and multiple future frames. Ifextrapolation is used for approximation, instead of interpolation, thenthe frame to be extrapolated may reside temporally after one or morereference frames or temporally before one or more reference frames. Someof the previous and future frames may produce better frame substitutionresults that others. Interpolation may be performed by any of a varietyof techniques, such as motion-compensated interpolation (MCI), linearinterpolation, bilinear interpolation, bicubic interpolation, splineinterpolation, nearest neighbor interpolation, non-linear interpolation,linear or non-linear filtering of candidate frames, or the like.Interpolation may make use of a single reference frame forunidirectional interpolation or two or more frames for bidirectionalinterpolation. Similarly, extrapolation may make use of a singlereference frame or two or more frames for unidirectional extrapolation.

FRUC unit 22 may analyze candidate previous and/or future frames andselect one or more of the frames for use as reference frames ininterpolation or extrapolation of the additional frame. FRUC unit 22 maybe configured to select particular reference frames that are more likelyto produce favorable frame substitution results. For example, FRUC unit22 may analyze one or more characteristics of a set of candidatereference frames. The candidate reference frames may be selected from asubset of frames within a video sequence. For example, FRUC unit 22 mayanalyze subsets of N previous frames and M future frames, where N and Mare either equal or not equal.

If analysis shows that none of the candidate reference frames has asufficient quality level for frame substitution, FRUC unit 22 maydisable frame substitution. In this case, FRUC unit 22 may detect apotential FRUC failure in terms of the quality of the frame substitutionresult. Instead of wasting power and computing resources to produce alow quality interpolation result, for example, FRUC unit 22 may applyframe repetition to produce an approximation of the skipped frame. Inthe case of frame repetition, as described previously, decoder 14 uses aduplicate version of one of the previous or future frames in place ofthe skipped frame. Hence, in a quality-focused mode, FRUC unit 22 may beconfigured to select particular reference frames for interpolation orextrapolation, and disable frame substitution when an acceptable qualitylevel is unlikely. FRUC unit 22 may seek higher quality levels tojustify frame substitution when decoder 14 operates in aresource-focused mode.

In some configurations, system 10 may provide one or more benefits. Forexample, in some configurations, system 10 may reduce power consumptionin video decoder 14 by disabling frame substitution when it is notlikely to produce favorable results. In addition, in someconfigurations, system 10 may enhance the quality of substituted framesby selecting particular reference frames for use in frame substitution.Selection of higher quality reference frames for use in interpolation orextrapolation of substituted frames may be useful in a variety of videoapplications, such as applications in which encoded video is compressedusing variable bit rate (VBR) rate control techniques. In the case ofVBR, quality may vary among different frames, such that some frames maybe better than other frames when used as reference frames forinterpolation or extrapolation.

Also, video decoder 14 may be configured to detect candidate referenceframes suffering slice, frame or block losses, and eliminate such framesfrom consideration when there is no error concealment mechanism applied,or if an available error concealment mechanism is not likely to providegood quality frames. Elimination of candidate reference frames withinsufficient error concealment may be useful when significanttransmission losses occur such as in the case of video telephonyapplications. A resource-focused mode may be useful in reducing powerconsumption while maintaining reasonable objective and subjective videoquality, e.g., in low-motion video clips.

FIG. 2A is a diagram illustrating an example of a simple technique forinterpolation of an additional frame to support a FRUC technique invideo decoder 14. In general, to interpolate a macroblock (MB) 28 in aninterpolated frame F_(t)′ between a selected previous frame F_(t−N) anda selected future frame F_(t+M), video decoder 14 may rely on a motionvector v_(NM) extending between an MB 30 in previous frame F_(t−N) and acorresponding MB 32 in future frame F_(t+M). In this example, time tindicates the temporal location, i.e., time, at which the additionalframe to be interpolated would appear in the video sequence. FramesF_(t−N) and F_(t+M) are frames that temporally precede (t−N) and follow(t+M), respectively, the additional frame F_(t)′ that requiresinterpolation. In the example of FIG. 2A, frames F_(t−N) and F_(t+M)serve as reference frames for interpolation of the additional frameF_(t)′.

N and M indicate temporal offsets, relative to time t, and may be equalor nonequal to one another. For example, if N=1 and M=2, frame F_(t−N)may be the frame immediately preceding the interpolated frame, and frameF_(t+M) may be the second frame after the interpolated frame. In asimple example in which N=1 and M=1, for interpolation, the vector v₁₃extending between frame F_(t−N) and frame F_(t+M) may generally bedivided by two (for 1:2 frame rate conversion) to produce motion vectorsv_(NM)/2 and −v_(MN)/2 and identify a corresponding MB 28 in the frameF_(t)′ to be interpolated. Hence, in this simplified example, theposition of MB 28 is a function of the motion vectors V_(NM)/2 and−v_(NM)/2, where N=1 and M=1 for purposes of this example. MB 28 may beassigned a set of pixel values that correspond to MB 30 or MB 32, or anaverage of the pixel values of MB 30 and 32. For higher or lower frameupconversion, e.g., 1:X conversion, the motion vectors would be scaledaccordingly. For other cases, e.g., in which at least one of N and M isnot equal to one, different motion vectors obtained through motionestimation and motion vector processing may be used.

In addition, for some types of interpolation, FRUC unit 22 may rely onmultiple reference frames, such as two or more previous frames and twoor more future frames. In general, a reference frame refers to a framethat is used, either alone or in combination with one or more otherreference frames, to interpolate a frame, such as a skipped frame. Inthe interpolation process, pixel values associated with macroblockspresent in one or more reference frames may be used to interpolate pixelvalues in corresponding macroblocks in the additional frame to beinterpolated, e.g., as shown in FIG. 2A. The pixel values may includeluminance and/or chrominance pixel values.

As one example, an interpolated macroblock may include pixel valuesequal to pixel values in a macroblock in a previous frame, pixel valuesin a macroblock in a future frame, or an average of the pixel values inthe corresponding macroblocks in the previous and future frames. Themacroblocks in the interpolated frame may be motion compensated relativeto corresponding blocks in the reference video frames, as shown in FIG.2A. The macroblocks may be identified by motion vectors extendingbetween the previous and future frames, as shown in FIG. 2A. Theillustration of interpolation shown in FIG. 2A is one example, andshould be considered nonlimiting of the techniques broadly described inthis disclosure. A wide variety of different interpolation techniquesmay be used for frame substitution in accordance with this disclosure.

FIG. 2B is a diagram illustrating interpolation of an extra video frameusing selected reference frames. In the example of FIG. 2B, FRUC unit 22selects reference frames F_(t−1) and F_(t+2) for use in interpolatingthe additional frame F_(t)′. FRUC unit 22 may analyze one or morecharacteristics of multiple previous frames F_(t−1), F_(t−2), andF_(t−3) and multiple future frames F_(t+1), F_(t+2), and F_(t+3). In theexample of FIG. 2B, FRUC unit 22 analyzes three previous referenceframes and three future reference frames for purpose of illustration. Inthis example, FRUC unit 22 may select one previous reference frame andone future reference frame, based on this analysis, for use ininterpolation of interpolated frame F_(t)′. However, the actual numberof previous and future reference frames may be different than theexample of FIG. 2B. In addition, the number of previous frames analyzedby FRUC unit 22 may be different from the number of future framesanalyzed by the FRUC unit. In general, FRUC unit 22 may select aprevious frame and a future frame that, based on the quality analysis,are likely to produce interpolation results with an acceptable level ofquality. In the example of FIG. 2B, selected reference frames F_(t−1)and F_(t+2) are indicated by cross-hatching.

FIG. 2C is a diagram illustrating an example of a technique forextrapolation of a video unit in video decoder 14. In the example ofFIG. 2C, two previous reference frames F_(t−M) and F_(t−N) are used toextrapolate an additional frame F_(t)′ to support frame substitution. Ingeneral, to extrapolate an MB 31 in a frame F_(t)′ following a selectedprevious frame F_(t−N) and a selected previous frame F_(t−M), videodecoder 14 may rely on a vector v extending between a corresponding MB33 in previous frame F_(t−N) and a corresponding MB 35 in previous frameF_(t−M). In this example, t indicates the temporal location, i.e., time,at which the additional frame to be interpolated would appear in thevideo sequence. Frames F_(t−N) and F_(t+M) are frames that temporallyprecede by (t−N) and (t+M), respectively, the additional frame F_(t)′that requires extrapolation. In the example of FIG. 2C, previousreference frames F_(t−N) and F_(t+M) serve as reference frames forextrapolation of the additional frame F_(t)′. However, one or moreprevious reference frames or one or more future reference frames may beused to extrapolate the additional frame F_(t)′. In other words, theadditional frame may be extrapolated forward or backward using previousframes or future frames, respectively.

As in the example of FIG. 2A, N and M in FIG. 2C indicate temporaloffsets, relative to time t, and may be equal or nonequal to oneanother. For example, if N=2 and M=1, frame F_(t−M) may be the frameimmediately preceding the extrapolated frame, and frame F_(t−N) may betwo frames before the extrapolated frame. MB 31 may be assigned a set ofpixel values that correspond to MB 33 or MB 35, or an average of thepixel values of MB 33 and 35. The extrapolation process may make use ofmotion compensated extrapolation. As in the case of interpolation, forextrapolation, a reference frame may refer to a frame that is used,either alone or in combination with one or more other reference frames,to extrapolate an extra frame to be added to the decoded video frames.

The extrapolation may be motion compensated by extrapolating motionvector v from corresponding blocks in the reference frames, as shown inFIG. 2C. In the extrapolation process, pixel values associated with MBspresent in one or more reference frames may be used to extrapolate pixelvalues in corresponding MBs in the additional frame to be extrapolated.The illustration of extrapolation shown in FIG. 2C is one example, andshould be considered nonlimiting of the techniques broadly described inthis disclosure. A wide variety of different extrapolation techniquesmay be used for frame substitution in accordance with this disclosure.

FIG. 2D is a diagram illustrating extrapolation of an extra video frameusing selected reference frames. In the example of FIG. 2D, FRUC unit 22selects reference frames F_(t−1) and F_(t−2) for use in extrapolatingthe additional frame F_(t)′. FRUC unit 22 may analyze one or morecharacteristics of multiple previous frames F_(t−1), F_(t−2), andF_(t−3) or multiple future frames, depending on whether previous orfuture frames are used for extrapolation. In the example of FIG. 2D,FRUC unit 22 analyzes four previous reference frames F_(t−1), F_(t−2),F_(t−3), F_(t−4) for purpose of illustration. In this example, FRUC unit22 may select two previous reference frames, based on this analysis, foruse in extrapolation of frame F_(t)′. However, the actual number ofreference frames used for extrapolation may be different than theexample of FIG. 2D. In general, FRUC unit 22 may select reference framesthat, based on the quality analysis, are likely to produce extrapolationresults with an acceptable level of quality. In FIG. 2D, selectedreference frames F_(t−1) and F_(t−2) are indicated by cross-hatching.

FIG. 3 is a block diagram illustrating an example of video decoder 14 ofFIG. 1 in more detail. In the example of FIG. 3, video decoder 14includes received frame buffer 34, decoding unit 36, frame substitutionunit 38, output frame buffer 40, FRUC analysis unit 42, and selectionunit 44. Frame substitution unit 38, FRUC analysis unit 42 and selectionunit 44 may form part of FRUC unit 22 of video decoder 14. In theexample of FIG. 3, FRUC unit 22 resides within video decoder 14. Asmentioned above, however, FRUC unit 22 may reside outside of videodecoder 14 in other implementations, e.g., within a video post-processormodule or video display processor or MDP device. Received frame buffer34 receives and stores encoded frames that are transmitted over channel19 from video encoder 12. Decoding unit 36 decodes the received framesusing an applicable coding process and places the decoded frames inoutput frame buffer 40.

The received frames 34 may exclude various frames to be interpolated orextrapolated. Such frames may include frames that were skipped byencoder 12, frames or portions of frames that are lost duringtransmission across channel 19, and frames that were not supported bythe basic frame rate of encoder 12. To promote spatio-temporal quality,frame substitution unit 38 may be configured to either interpolate orextrapolate additional frames based on analysis and selection ofparticular received frames for use in the interpolation orextrapolation, as applicable.

As mentioned previously, interpolation by frame substitution unit 38 mayinclude any of a variety of interpolation techniques such asmotion-compensated interpolation (MCI), linear interpolation, bilinearinterpolation, bicubic interpolation, spline interpolation, nearestneighbor interpolation, or the like. Interpolation may make use of asingle reference frame for unidirectional interpolation or two or moreframes for bidirectional interpolation. Likewise, extrapolation may relyon one or more frames. In some cases, frame substitution unit 38 maydisable frame substitution and instead apply frame repetition, e.g., byrepeating a previous or future frame in place of a frame to be added.

Frame substitution unit 38 adds substituted or repeated frames to videooutput frame buffer 40. The decoded frames and substituted or repeatedframes in video output frame buffer 40 may be used to drive a videooutput device such as a display. As an example, video decoder 14 mayform part of any of a variety of devices including digital videocapabilities, including a wireless communication device such as a mobileradio telephone, a digital media player, a personal digital assistant(PDA), a digital television, or the like. Alternatively, the frames inoutput frame buffer 40 may be transmitted to one or more other devicesfor archival or display. In each case, the substituted or repeatedframes produced by frame substitution unit 38 supplement frames decodedby decoding unit 36, e.g., to enhance temporal visual quality of a videoclip.

As further shown in FIG. 3, frame substitution unit 38 may receivedecoded frames from the output of decoding unit 36 for use as referenceframes in interpolating or extrapolating frames. The decoded framesreceived from decoding unit 26 may be pixel domain frames produced bydecoding unit 36 based on encoded frames from received frame buffer 34.Frame substitution unit 38 may use the decoded reference frames asreference frames for interpolation or extrapolation of additionalframes. The particular reference frames used for interpolation orextrapolation may be identified by selection unit 44 based on analysisof candidate reference frames by FRUC analysis unit 42.

Analysis unit 42 may be provided to analyze candidate reference videoframes. For example, analysis unit 42 may obtain decoded frames in thepixel domain from the output of decoding unit 36, e.g., for analysis ofobjective quality metrics such as a structural similarity metric (SSIM),blockiness, and/or blurriness, and optionally color bleeding, todetermine the quality of candidate reference frames for use ininterpolation or extrapolation. Alternatively, or additionally, analysisunit 42 may analyze coding information for candidate reference frames,such as QP values and CBP values associated with the candidate frames,as an indication of the relative quality levels of the frames. Codinginformation such as QP values and CBP values may be parsed from thebitstream associated with frames in received frame buffer 34, e.g., bydecoding unit 36, and provided to analysis unit 42 as further shown inFIG. 2A. Hence, in some implementations, analysis unit 42 may receivedecoded pixel domain frames from the output of decoding unit 36 and/orbitstream information such as QP and CBP values parsed from candidateframes by decoding unit 36.

The video frames obtained from decoding unit 36 may be consideredcandidate reference frames in the sense that they may be used bysubstitution unit 38 as reference frames for substitution of additionalframes. For each frame to be added, analysis unit 42 may analyze pixeldomain information and/or bitstream information for a subset of previousand/or future frames, relative to the frame to be added, and provide anoutput to selection unit 44 to identify frames that should be selectedfor use in frame interpolation or extrapolation by substitution unit 38.Selection unit 44 may be configured to select one or more frames forinterpolation or extrapolation of the additional frame based on theanalysis output by analysis unit 42. In addition, in some cases,selection unit 44 may be configured to direct substitution unit 38 toenable or disable frame substitution, e.g., when the analysis indicatesthat none of the candidate reference frames is suitable for use in framesubstitution at an acceptable quality level.

As shown in FIG. 3, selection unit 44 may generate a frame selectionsignal or command that directs frame substitution unit 38 to select oneor more of the frames from received frame buffer 34 for use ininterpolation or extrapolation of the frame to be added. For example,for each frame to be added, selection unit 44 may direct framesubstitution unit 38 to select one previous frame and one future framefor use in interpolation. Frame substitution unit 38 then may apply theselected frames as reference frames for interpolation of a skippedframe. As an illustration, a previous frame and a future frame may beused to interpolate blocks in the frame to be added based on motionvectors extending between corresponding blocks in the previous andfuture frames, e.g., as shown in FIG. 2A. As another illustration,selection unit 44 may direct frame substitution unit 38 to select a pairof previous frames for use in extrapolation of the additional frame.

Selection unit 44 also may generate a frame substitution enable/disablecommand. When the analysis by analysis unit 42 indicates that none ofthe candidate reference frames is suitable for use in framesubstitution, selection unit 44 may disable interpolation orextrapolation by frame substitution unit 38. Also, selection unit 44 maydisable interpolation in a resource-focused mode when the visualdifference between frame substitution and frame repetition is relativelyimperceptible. In either case, frame substitution unit 48 may repeat aprevious or future frame in place of the additional frame instead ofperforming interpolation or extrapolation to produce a substitute frame.Frame repetition may consume substantially less power and computingresources than interpolation or extrapolation. When analysis unit 42indicates that received frame buffer 34 contains suitable referenceframes, selection unit 44 may enable frame substitution and identify theselected frames for use in the frame substitution.

The reference frames selected for frame substitution may be the framesclosest to the frame to be added or frames further away from the frameto be added. In some cases, even though they would typically be selectedfor an ordinary FRUC process, the closest frames (i.e., nearest previousand nearest future frames) may actually have characteristics, e.g., dueto VBR coding or other factors, that make them less suitable for use ininterpolation or extrapolation than other reference frames. Analysisunit 42 analyzes the candidate reference frames to provide an indicationof the suitability of such frames for use in frame substitution. Insteadof simply using frames adjacent to the skipped frame as a matter ofdefault, analysis unit 42 and selection unit 44 may permit framesubstitution unit 38 to use reference frames that provide more optimalresults.

FIG. 4 is a block diagram illustrating another example of video decoder14 of FIG. 1 in more detail. In the example of FIG. 4, video decoder 14substantially corresponds to the video decoder of FIG. 3A. FRUC unit 22of video decoder 14 of FIG. 4 further includes, however, a modeselection unit 46 and a resource monitor 48. In addition, FRUC unit 22may optionally include a delay detection unit 51. Mode selection unit 46may support two or more modes of operation of FRUC unit 22. For example,FRUC unit 22 may be configured to operate in a quality-focused mode as afirst operational mode or a resource-focused mode as a secondoperational mode. In the example of FIG. 4, FRUC unit 22 resides withinvideo decoder 14. In other implementations, FRUC unit 22 may resideoutside of video decoder 14, e.g., within a video post-processor moduleor video display processor or mobile display processor.

Resource monitor 48 may be configured to monitor, detect, estimate orotherwise determine available power, computing and/or memory resourceswithin a device in which video decoder 14 is provided. In some cases,resource monitor 48 may be configured to monitor resource budgetsapplicable to the processing of frames. Hence, resource monitor 48 maybe configured to monitor actual resources available at a given time thata frame is processed within video decoder 14, or monitor estimatedresource consumption relative to a resource budget applicable toprocessing of a frame. In response to different resource levels,resource monitor 48 may trigger mode selection unit 46 to select adifferent mode (e.g., quality-focused or resource-focused) for operationof FRUC unit 22. Mode selection unit 46 may transmit a mode selection toFRUC analysis unit 42 to modify its analysis of candidate frames forreference frame selection. Alternatively, mode selection unit 46 maytransmit a mode selection to enable or disable substitution (e.g.,interpolation or extrapolation), as will be described.

As an example, resource monitor 48 may be configured to determine apower level by monitoring or estimating consumption of processingresources in decoder 14 relative to a budget for consumption ofprocessing resources. In general, there is a correspondence between MIPS(million instructions per second) spent in a processor such as a digitalsignal processor (DSP) and power consumed for the DSP operations. Thereis also a correspondence between the amount of data fetched fromexternal memory and power consumed for fetching such data. In addition,there is a correspondence between the amount of frame data sent to adisplay and power spent for this operation. This correspondence can bereliably established for a known device or chipset and then representedby entries in a lookup table. For example, MIPS, data fetching amounts,and display amounts may be used as indices that map to power consumptionvalue entries in a lookup table.

For a FRUC application in a given chipset, it is possible to determinehow much each operation requires in terms of MIPS, how much data isfetched from the external memory, as well as how much data is sent to adisplay. In one scenario, resource monitor 48 may be configured tocalculate the power consumed for generation and display of eachinterpolated or extrapolated frame, and compare the level of consumedpower to a power budget allocated to a frame or set of frames. The powerbudget may be a predetermined, fixed or adjustable level of powerspecified as a design requirement for a functional unit of a device,such as a video decoder (or entire CODEC) in a mobile wireless handsetor other device.

The power budget may be allocated for a series of frames, such as agroup of pictures (GOP) or other video sequence. As frames areprocessed, and power is consumed, the available power in the powerbudget is reduced. Resource monitor 48 may be configured to estimate howmuch power will be required to interpolate or extrapolate a new FRUCframe, e.g., based on a known correspondence between MIPS, datafetching, and display of an interpolated or extrapolated frame, whichmay be stored in a lookup table. As mentioned above, the power budgetmay be either fixed (i.e., not adjusted by a feedback loop from thedevice), or it can be adjusted based on a feedback loop from the device.

For a given frame, if there is enough power remaining in the powerbudget for interpolation, the interpolation process can be enabled.Alternatively, the interpolation process can be disabled if there isinsufficient power remaining in the power budget. In accordance with anaspect of this disclosure, instead of enabling and disablinginterpolation, resource monitor 48 and mode selection unit 46 may beconfigured to select different interpolation/extrapolation modes. Inparticular, based on an available resource level determined by resourcemonitor 48, mode selection unit 46 may select either the quality-focusedmode or the resource-focused mode.

For example, if MIPS, data fetching and/or display of previous frames ina video sequence have reduced the available power budget to a levelbelow a first predetermined threshold, resource monitor 48 may indicateto mode selection unit 46 that the resource-focused mode should beselected. Alternatively, if the available power budget is above thefirst predetermined threshold, resource monitor 48 may indicate to modeselection unit 46 that the quality-focused mode should be selected. Insome aspects, if the available power budget is below a secondpredetermined threshold, which is less than the first predeterminedthreshold, resource monitor 48 may indicate thatinterpolation-extrapolation could be disabled in favor of framerepetition or, possibly, neither interpolation, extrapolation nor framerepetition should be enabled, in order to conserve power in the device.

In monitoring available power in the power budget, resource monitor 48may track MIPS expended per frame, data fetched per frame, and theamount of frame data sent for display over a given video sequence. Forexample, resource monitor 48 may maintain a running, cumulative total ofMIPS, fetched data and displayed data for a particular GOP or othervideo sequence, and map such values to corresponding power consumptionvalues. In particular, resource monitor 48 may map each of the values toa corresponding power consumption value, e.g., in one or more lookuptables, and then sum the values to produce an overall power consumptionvalue. As an alternative, resource monitor 48 may access amulti-dimensional lookup table that maps a combination of indicesrelating to MIPS, fetched data and displayed data to an overall powerconsumption value for a frame or other video unit.

As a further alternative, resource monitor 48 may map such values topower consumption values on a frame-by-frame basis, and cumulatively addthe power consumption values to produce a running total of powerconsumption over a video sequence. As an illustration, if a videosequence (e.g., a sequence of thirty frames) has a power budget of Xand, at the nth frame in the sequence, the MIPS, fetched data anddisplayed data are estimated to have consumed an amount of power Y suchthat the available power budget is X−Y=Z, then resource monitor 48 maycompare the available power budget to the predetermined first thresholdto select either a quality-focused mode or a power-focused mode.

The power budget for the sequence may be divided by the number of framesin the sequence and updated over the course of the sequence in a varietyof ways to provide more even allocation over the entire sequence, orframes in the sequence that may justify additional power budget, so thatthe power budget is not expended unevenly earlier in the video sequence.Various alternatives are available for estimating power consumptionrelative to a power budget in order to support selection of aquality-focused mode or a power-focused mode. Accordingly, the aboveexamples are provided for purposes of illustration, and should not beconsidering limiting of the techniques broadly described in thisdisclosure.

In a quality-focused mode, FRUC unit 22 may perform reference frameselection based exclusively, primarily or substantially on one or morereference frame quality criteria. The quality criteria may indicate, forexample, a level of interpolation or extrapolation quality likely to beproduced by a selected reference frame. If none of the reference framessatisfies the quality criteria, frame substitution may be disabled byselection unit 44 for a particular frame to be added. In this case,frame substitution unit 38 may apply frame repetition instead of frameinterpolation to approximate the frame to be added.

In a resource-focused mode, FRUC unit 22 may selectively enable ordisable frame substitution for some frames based on a combination ofboth resource and quality considerations. For example, selection unit 44may disable frame substitution when FRUC analysis unit 42 determinesthat the video sequence is generally static in the vicinity of the frameto be substituted. If the video sequence contains a low level of motion,there may be little or no advantage in performing substitution becausethe visual differences between interpolation or extrapolation and framerepetition may be small or even imperceptible. In this case, bydisabling substitution, decoder 14 can avoid expending power andresources for interpolation or extrapolation with little impact tovisual quality.

When there is sufficient motion to justify substitution in theresource-focused mode, analysis unit 42 may perform the quality analysisidentical or similar to that used in the quality-focused mode. Hence,the resource-focused and quality-focused modes may not be completelyseparate modes. Rather, when the resource-focused mode is activated, thequality-focused mode may proceed only if the motion level is sufficientto justify interpolation or extrapolation, as applicable. Thequality-focused mode may be activated alone, or the resource-focusedmode may operate to disable frame substitution and thereby deactivatethe quality-focused mode or enable substitution and activate thequality-focused mode. Notably, even in the quality-focused mode,substitution may be disabled if none of the candidate reference framessatisfy the quality criteria.

As an option, in the resource-focused mode, when substitution isenabled, the quality thresholds used for the quality analysis may beadjustable, e.g., as a function of the resource saving requirements ofthe video decoder 14. For example, the threshold may be adjusted basedon available interpolation resources of the video decoder 14, such asavailable power, computing resources, and/or memory resources. In someimplementations, the quality threshold may be adjusted based on a levelof available power resources, such as a level of available powerresources associated with video decoder 14, a video post-processor,and/or a video display processor, such as a mobile display processor(MDP). In either case, whether thresholds are fixed or adjustable, ifsubstitution is enabled, analysis unit 42 may apply one or more qualitycriteria to select reference frames or disable substitution.

As an example, if the video sequence in the vicinity of the frame to beadded is characterized by very low motion video content, the benefit ofvideo frame interpolation or extrapolation may not be very noticeable.Hence, FRUC unit 22 may decide not only which frames to use as referenceframes for video frame substitution, but also which frames to add byframe substitution, i.e., which frames to interpolate or extrapolate.For some frames, the cost of interpolation or extrapolation may not bejustified by sufficient enhancement of temporal visual quality. Forexample, by avoiding interpolation for some skipped frames, in theresource-focused mode, FRUC unit 22 may balance savings in computingresources and associated power consumption with enhancement ininterpolated frame quality and temporal visual quality of the videosequence. A zero motion vector count and/or small motion vector countmay be used as decision criteria for determining whether to interpolateor extrapolate a particular frame based on motion content. Countthresholds may be derived in different ways. For example, the thresholdsmay be fixed for one or both of the zero motion vector count and thesmall motion vector count judge motion activity. Alternatively, one orboth of the thresholds may be adjusted, e.g., based on resource levelsof decoder 14.

In the examples of FIGS. 3 and 4, for quality-focused selection ofreference frames for substitution, video decoder 14 may rely on avariety of quality-related criteria. For example, FRUC unit 22 mayselect reference frames for use in video frame substitution based onslice and/or frame loss information of the reference frames incombination with the reliability and type of error concealment methodprovided to reconstruct the reference frames. FRUC unit 22 may beconfigured to analyze the level of error due to the loss, and thequality of the error concealment mechanism available to correct theerror. Alternatively, or additionally, FRUC unit 22 may analyze QPvalues and CBP values associated with the candidate frames as anindication of the relative quality levels of the frames. For example,the QP value may be coupled with the CBP value to judge the quality of areference frame.

FRUC unit 22 also may apply objective visual quality metrics. Theobjective visual spatial quality metrics may be non-reference metricssuch as structural similarity metric (SSIM), blockiness, and/orblurriness. The objective quality metrics may alternatively oradditionally include color bleeding. The objective quality metrics maybe used to produce quality scores for the candidate reference frames.Other examples of quality criteria may include the types of intra modesused in the candidate reference frames, or the intra mode and motionvector count for each of the candidate reference frames. Additionalcriteria may also be utilized as needed.

In some aspects, as described previously, FRUC unit 22 may be configuredto disable frame substitution or bias the selection of reference videoframes toward previous frames instead of future frames when a givenvideo application requires shorter delays in end-to-end processing. In avideo telephony application, for example, users may be engaged in areal-time or near real-time video conference which may require reductionor elimination of delays in processing and presentation of video. If aframe substitution technique relies on future frames, e.g., forinterpolation or extrapolation, the delay in decoding and processingsuch future frames in order to produce an additional frame may not betolerable. In this case, when the video application imposes a delayrequirement, FRUC unit 22 may be configured to disable framesubstitution, prohibit selection of future candidate reference frames,or prohibit selection of future candidate reference frames that are notwithin a predetermined number of frames of the frame to be added.

For example, as shown in FIG. 4, FRUC unit 22 may optionally include adelay detection unit 51 that detects a maximum delay or other quality ofservice requirement imposed by a given video application, and directsselection unit 44 to disable frame substitution or require selection ofpast frames or early future frames as reference frames for interpolationor extrapolation. If delay detection unit 51 detects that a videotelephony application or other delay-sensitive video applicationrequires minimal delay, the delay detection unit 51 may direct selectionunit 44 to disable frame substitution. A delay-sensitive application ortype of application that may be delay-sensitive may be detected by delaydetection unit 51 based on receipt of a specific signal from a device inwhich a decoder 14 is embedded, receipt of a specific signal provided inside information associated with video data received by the decoder 14,or in any of a variety of other ways. In the case of detection of adelay-sensitive application, substitution unit 38 may apply framerepetition or deactivate FRUC altogether in order to avoid disconcertingdelays that could undermine the quality of a video conference for theusers.

Alternatively, delay detection unit 51 may direct selection unit 44(and/or analysis unit 42) to enable frame substitution but require theuse of past reference frames or early future reference frames forinterpolation or extrapolation, e.g., based on a temporal distancerequirement. Hence, delay detection unit 51 may impose constraints onselection unit 44 to avoid selection of particular future referenceframes indicated as having sufficient quality, but excessive temporaldistance from the frame to be added.

For example, frames that are one or two frames into the future, relativeto the frame to be added, may be acceptable for use as reference framesfor frame substitution. If a future frame is at a temporal distance thatis several frames away from the frame to be added, however, theprocessing and presentation delay that could result from waiting for theframe to be decoded and analyzed may be unacceptable. The distanceanalysis may be applied to candidate reference frames after qualityanalysis or before quality analysis. If the distance analysis is appliedbefore quality analysis, delay detection unit 51 may direct analysisunit 42 to suspend the quality analysis for a particular candidatereference frame if the candidate reference frame is a future frame andthe distance of the frame from the frame to be added is too great.

As an alternative, delay detection unit 51 may direct analysis unit 42to adjust its quality analysis when delay is a concern such that futurereference frames that are situated an excessive distance from the frameto be added are given lower quality scores, or excluded fromconsideration in the quality analysis. In either case, the effect may bethat some future reference frames are excluded from selection asreference frames so that frame substitution may proceed withoutadversely impacting delay characteristics of a delay-sensitive videoapplication, such as video telephony. In some cases, for video telephonyor other delay-sensitive video applications, delay requirements maypreclude frame substitution, e.g., due to the real-time or nearreal-time service requirements of the video telephony application. Forvideo streaming and playback, however, delay issues ordinarily may beless of a concern.

As described above, in some implementations, one or more of thecandidate reference video units for use in interpolation orextrapolation of an additional video unit may be selected based at leastin part on the temporal distance of one or more of the candidatereference video units from the additional video frame. The candidatereference video units may be selected based on temporal distance when adelay-sensitive application is detected. Alternatively, the candidatereference video units may be selected based on temporal distance on aregular basis, or in response to some other triggering event. Hence, insome aspects, temporal distance may form part of a quality analysis thatis applied for selection of reference frames for use in interpolation orextrapolation.

FIG. 5 is a block diagram illustrating a reference frame analysis unit42 for use with the video decoder 14 of FIG. 3 or FIG. 4. As mentionedpreviously, although analysis and selection of reference video frames isdescribed for purposes of illustration, the structure and functionalityof analysis unit 42 may be adapted for analysis and selection of otherreference video units, such as slices or blocks (e.g., macroblocks orsmaller blocks). In the example of FIG. 5, analysis unit 42 includes anobjective metric checker 50, an error concealment (EC) checker 52, aquantization parameter (QP) checker 54, a coded block pattern (CBP)checker 56, a quality score calculator 58, compare unit 59, and a motionvector (MV) reliability checker 60.

The various units shown in FIG. 5 may be used in a quality-focused modeof operation, as well as a resource-focused mode of operation wheninterpolation is enabled. When the resource-focused mode of operation isselected, e.g., by mode selection unit 46, analysis unit 42 may activatea motion analyzer 64. In addition, optionally, analysis unit 42 mayactivate a mode adjustment unit 62 in the resource-focused mode.Selection unit 44 may consider outputs of MV reliability checker 60 andmotion analyzer 64 to determine whether to perform frame substitutionfor a frame to be added and, if so, whether to select a frame beinganalyzed as a reference frame for use in frame substitution to add theframe.

Objective metric checker 50 may be configured to analyze candidatereference frames to determine SSIM values, and/or degrees of blockiness,blurriness, or color bleeding associated with such frames, and generatea quality score based on the determination. The quality score producedby objective metric checker 50 for a particular candidate referenceframe may be low when substantial blockiness, blurriness, and/or colorbleeding is detected, and high when substantially no blockiness,blurriness, and/or color bleeding. The quality score for differentcandidate reference blocks may vary between high and low as a functionof such objective visual quality metric characteristics. Alternatively,the quality score may be expressed as either high or low based oncomparison to a predetermined threshold.

When analysis unit 42 makes use of objective metric checker 50, theobjective metric checker may be applied to decoded frames that arereconstructed by decoding unit 36. Hence, analysis unit 42 may analyzeencoded frames obtained from received frame buffer 34 and, via objectivemetric checker 50, may receive and analyze reconstructed frames obtainedby decoding frames from received frame buffer 34 via decoding unit 36.Objective metric checker 50 may analyze the reconstructed candidatereference frames for SSIM value, blockiness, blurriness, color bleedingor other objective quality metrics to generate a quality score.

EC checker 52 analyzes the candidate reference frames to determinewhether each frame has suffered slice and/or frame loss and, if so,whether a reliable error concealment mechanism is available toreconstruct the encoded frame obtained from received frame buffer 34. Inthis sense, EC checker 52 may operate as an EC reliability checker. Ifthe loss results in an error above a predetermined level, EC checker 52may evaluate the quality of the error concealment mechanism available tocorrect the error. When there is no slice or frame loss, or there is asufficient error concealment mechanism to compensate for the slice orframe loss, EC checker 52 may generate a relatively high score for aparticular candidate reference frame. When error concealment isinsufficient to compensate for the slice or frame loss, EC checker 52may generate a low score. The score generated by EC checker 52 may varyaccording to the amount of slice or frame loss, or simply a high or lowscore based on comparison to a predetermined threshold.

EC checker 52 may determine whether there is a sufficient errorconcealment mechanism available in a variety of ways. As an example, ECchecker 52 may identify the type of error concealment based onpredetermined knowledge of the error concealment used for the type ofvideo coding, e.g., H.264 (using motion vector from co-locatedmacroblock), or the type of video coding known to be performed in thechip or chipset in which the decoder 14 is implemented. EC checker 52may rely on previous off-line analysis of the effectiveness ofparticular error concealment mechanisms for different slice or frameloss scenarios.

For example, each error concealment mechanism may be rated foreffectiveness in dealing with different numbers of losses, differentlocations of losses, numbers of affected blocks or units, errordetection time, and the like. EC checker 52 may determine one of more ofthe above characteristics (i.e., number of losses, locations, errordetection time) associated with a particular slice or frame loss, andthen determine whether the available error concealment mechanism will beeffective given those characteristics, and given prior knowledge of theperformance of such an error concealment mechanism under similarconditions.

As an illustration, if error concealment mechanism X is used in decoder14, and it is known that mechanism X is effective for lossescharacterized by number of losses less than Y1, location of losses inparticular regions, and number of affected blocks/units less than Y2,given time for error detection processing of more than a time Z, then aloss that conforms to the above characteristics should be capable ofconcealment using mechanism X, in which case EC checker 52 may generatea high EC score.

In various scenarios, error concealment may fail if any one of thecharacteristics is not satisfied, or if one or more combinations ofcharacteristics present in the slice or frame loss. If the loss does notconform to the above characteristics, EC checker 52 may generate a lowEC score. Hence, EC checker 52 may rely on knowledge of the type oferror concealment mechanism, and its effectiveness in concealing sliceor frame losses having different characteristics.

QP checker 54 analyzes the quantization parameter (QP) values associatedwith each candidate reference frame. In general, a QP value indicates aquantization step size of the transform coefficients in a respectiveblock of an encoded video frame. The QP value may be the same for allblocks in a frame, or vary for different blocks. As an example, QPchecker 54 may analyze an average QP value of the blocks that form areference frame. Alternatively, QP checker 54 may analyze a maximum orminimum QP value for a frame. QP checker 54 may generate a relativelylow score if the average QP value for a frame is high, indicating coarsequantization of the video data in case of H.264 coding. Smaller QPvalues indicate finder quantization step size in H.264 coding, andgenerally higher video quality. Hence, for H.264 coding, if the averageQP value is lower, QP checker 54 may generate a higher quality score.

CBP checker 56 analyzes a CBP value for each candidate reference frame.The CBP value may be an average CBP value for the blocks that form theframe. In general, the CBP value indicates that there are either nononzero transform coefficients in a block or that there is at least onenonzero coefficient in a block. The transform coefficients may reside ina compression domain such as a discrete cosine transform (DCT), wavelettransform, or other compression transform domain. The CBP value analyzedby CBP checker 56 may be a luma or chroma CBP value, or both. For agiven frame, CBP checker 56 may determine the number of blocks for whichthe CBP value indicates at least one nonzero transform coefficientvalue. In some aspects, QP checker 54 and CBP checker 56 may becombined. In this case, a combined score may be generated based on theaverage QP value and the CBP value.

If the QP value is below a predetermined QP threshold, and the CBP valueis generally zero, indicating that substantially none of the blocks havenonzero coefficients, the quality of the frame may be indicated by ahigh score. A CBP value that is generally regarded as zero may bedetermined when the number of blocks for which the CBP value indicatesat least one nonzero coefficient is below a CBP threshold.Alternatively, if the QP value is above the QP threshold, then the scoremay be low. A very high QP and zero or medium CBP value may indicate alow score. If the CBP is zero and the QP value is below the threshold,the motion estimation for a block is very accurate, which should resultin a high quality score.

Quality score calculator 58 may be configured to generate an overallquality score for a candidate reference frame based on the outputs ofone or more of objective metric checker 50, EC checker 52, QP checker54, and CBP checker 56. The individual scores produced by objectivemetric checker 50, EC checker 52, QP checker 54, and CBP checker 56 maybe constructed simply as high or low quality values, or high, medium orlow quality values. Alternatively, the scores may be computed alongnumerous gradations or a substantially continuous scale. Quality scorecalculator 58 may accord equal weighting to each of the scores producedby objective metric checker 50, EC checker 52, QP checker 54 and CBPchecker 56. Alternatively, quality score calculator 58 may apply apredetermined, uneven weighting to the scores such that the totalquality score is a weighted summation of all of the scores.

As an illustration, in some implementations, quality score calculator 58may attribute more weight to the scores output by QP checker 54 and CBPchecker 56, than to the scores output by EC checker 52 and objectivemetric checker 50. In other implementations, the output of EC checker 52may be more important. As stated above, quality score calculator 58 mayweight the outputs. As an alternative, each of the checkers 50, 52, 54,56 may be configured to produce an individually weighted score withpreassigned weights. Quality score calculator 58 outputs a total scoreindicative of the quality of the candidate reference frame for use ininterpolation of a skipped frame.

Given the quality characteristics used to calculate the score, e.g.,objective metrics, EC characteristics, QP and CBP characteristics, thetotal score may indicate whether the candidate reference frame is likelyto produce a frame substitution result with an acceptable level ofquality. Compare unit 59 may compare the total score to a qualitythreshold value. If the total score is satisfactory, e.g., exceeds thequality threshold value, compare unit 59 indicates that the candidatereference frame has a level of quality that is acceptable for selectionas a reference frame for interpolation or extrapolation, as applicable,of an additional, i.e., extra, frame. If the total score is less thanthe quality threshold value, then compare unit 59 determines that thecandidate reference frame is not acceptable for selection as a referenceframe for interpolation or extrapolation, as applicable. In each case,analysis unit 42 may then proceed to analyze the next candidatereference frame for the additional frame currently under consideration,or proceed to analyze reference frames for the next frame to be added.The number of candidate reference frames considered for a particularsubstitute frame may be varied according to design considerations.

If the total score is satisfactory, e.g., meets or exceeds the qualitythreshold value, compare unit 59 may indicate that the candidatereference frame under consideration is suitable for selection as areference frame on the basis of quality. Alternatively, compare unit 59may select the reference frames that are ranked most highly, asdescribed above. In either case, in some implementations, to approve thereference frame for selection, analysis unit 42 may further include amotion vector (MV) reliability checker 60. MV reliability checker 60 mayanalyze the reliability of motion vectors in the candidate referenceframe to ensure that the selected reference frames will produce qualityframe substitution results if video frame substitution makes use of amotion-compensated prediction method for interpolation or extrapolation.

If the motion vectors are reliable, MV reliability checker 60 mayindicate to selection unit 44 that the candidate reference frame underanalysis may be selected as a reference frame for frame substitution. Ifthe motion vectors are not reliable, however, MV reliability checker 60may reject the candidate reference frame even though it has satisfiedthe quality requirements of compare unit 59. In this case, MVreliability checker 60 may indicate to selection unit 44 that thecandidate reference frame under analysis should not be selected as areference frame for frame substitution.

In some implementations, compare unit 59 may indicate appropriate framesfor selection as reference frames as they are considered. In otherwords, as analysis unit 42 analyzes each candidate reference frame, ifit identifies a suitable candidate reference frame, it may indicate thatthe candidate reference frame should be selected. The process maycontinue until analysis unit 42 identifies a sufficient number and typeof candidate reference frames, at which point analysis unit 42 may stopanalysis of candidate reference frames for interpolation orextrapolation of a current frame and move on to analysis of candidatereference frames for interpolation of the next frame to be added in thevideo sequence.

As a simple illustration, analysis unit 42 may identify a singleprevious reference frame and a single future reference frame forinterpolation of a frame that resides temporally between the previousand future frames. Alternatively, for more complex types ofinterpolation, analysis unit 42 may identify multiple previous andfuture reference frames that can be selected for interpolation of aframe to be added. In each case, analysis unit 42 may analyze a subsetof previous and future frames adjacent to the frame to be added until asufficient number and type (e.g., previous and future, if necessary) areidentified for selection. A finite number of frames may be analyzed. Ifnone of the analyzed frames produces a sufficient quality score, thenanalysis unit 42 may indicate that no frames are selected, and thatframe substitution unit 38 should apply frame repetition instead offrame substitution.

In some implementations, analysis unit 42 may use other types of qualitycriteria, such as the types of intra-coding modes used in candidatereference frames and/or intra mode and motion vector counts for thecandidate reference frames. As an example, if the number of intra-codedblocks (e.g., macroblocks) in a particular candidate reference frameexceeds a mode decision threshold, analysis unit 42 may indicate that alow interpolation quality for the reference frame, e.g., such that thecandidate reference frame should not be used for interpolation of anadditional frame. The mode decision threshold may be static ordynamically adjusted, e.g., based on a number of blocks having a numberof coding bits above a threshold in a candidate reference frame. A widevariety of other quality criteria may be used. Such information used forquality analysis may be considered as alternatives, or in addition, toother information described in this disclosure, such as objectivequality metrics, QP and CBP characteristics, and EC characteristics.

Some additional frames may be interpolated or extrapolated using some ofthe same reference frames as other nearby frames to be interpolated orextrapolated. For this reason, in some cases, once a candidate referenceframe is analyzed for quality, it may be desirable to store informationrelating to the quality for the candidate reference frame. In thismanner, if the particular candidate reference frame is later consideredas a candidate reference frame, its quality can be quickly determinedwithout the need to perform the analysis again. The candidate referenceframe may be relevant as a potential reference frame for a few frames tobe interpolated or extrapolated in a portion of a video sequence.

As the video sequence proceeds, the stored information may becomeobsolete or at least less relevant due to the increasing temporalremoteness of the candidate reference frame relative to the frames to beinterpolated or extrapolated. Accordingly, the information can bediscarded at some point, e.g., after the video sequence has proceed to apoint that is more than a predetermined number of frame away from thecandidate reference frame. By storing quality information, it may benecessary to perform the quality analysis in analysis unit 42 only oncefor each candidate reference frame. Alternatively, quality analysis maybe performed each time the frame is identified as a candidate referenceframe for a frame to be added.

In some implementations, analysis unit 42 may be configured to rank thecandidate reference frames that are analyzed in terms of pertinentquality levels. For example, analysis unit 42 may include a ranking unit(67A or 67B) that ranks candidate reference frames. In this case,instead of stopping when a suitable number and type of frames havingquality scores that satisfy the quality threshold value have been found,analysis unit 42 may rank candidate reference frames to identify theframes producing the best quality scores. For example, analysis unit 42may rank candidate reference frames that satisfy a quality level, e.g.,meet or exceed the quality threshold value, as indicated by compare unit59, in order of total quality score, and select the candidate referenceframes that are ranked most highly.

As one option, with reference to FIG. 5, analysis unit 42 may include aranking unit 67A that evaluates and ranks candidate reference framesidentified by compare unit 59 as satisfactory, i.e., as meeting orexceeding the quality threshold value. Ranking unit 67 may select anumber of the most highly ranked reference frames, consistent with thenumber of reference frames required for the type of frame substitutionapplied by frame substitution unit 38. The selected, i.e., most highlyranked candidate reference frames, then may be passed from ranking unit67A to MV reliability checker 60 to determine whether they have reliableMV content. If so, MV reliability checker 60 may indicate to selectionunit 44 that the frames should be selected for use in framesubstitution. If the most highly ranked candidate reference frames donot have reliable MV content, MV reliability checker 60 indicates toselection unit 44 that frame substitution unit 38 should apply framerepetition instead of frame substitution.

As another option, because the ranking may be influenced by unreliableMV content, analysis unit 42 may pass all ranked candidate referenceframes having quality scores that are satisfactory, e.g., meet or exceedthe quality threshold value, to MV reliability checker 60. Inparticular, compare unit 59 may provide all candidate reference frameswith passing scores to MV reliability checker 60. MV reliability checker60 identifies the candidate reference frames that have both asatisfactory quality score, e.g., that meets or exceeds the qualitythreshold value, e.g., as indicated by compare unit 59, and reliable MVcontent. Analysis unit 42 may optionally include a ranking unit 67B thatreceives the output of MV reliability checker 60 and ranks the candidatereference frames in terms of quality score. Ranking unit 67B then mayselect a number of the most highly ranked candidate reference frames,and communicate the selected reference frames to selection unit 44.Selection unit 44 may notify frame substitution unit 38 of the selectedreference frames for use in frame substitution to add a video frame.

MV reliability checker 60 may omit candidate reference frames that donot have reliable MV content, leaving the remaining candidate referenceframes for ranking and selection as reference frames by ranking unit67B. In particular, ranking unit 67B may identify the most highly rankedcandidate reference frames that have reliable MV content to selectionunit 44 for selection as reference frames as interpolation. Again, thenumber of reference frames that are selected may be a function of thetype of frame substitution used by frame substitution unit 38, i.e., thetype of interpolation or extrapolation, and the number of referenceframes needed to support the frame substitution process.

MV reliability checker 60 may analyze motion vector reliability usingany of a variety of different techniques. As one example, MV reliabilitychecker 60 may be configured to apply a motion difference basedapproach. MV reliability checker 60 may operate to analyze motion vectorinformation in both an X (horizontal) direction and a Y (vertical)direction of a video frame. In this case, MV reliability checker 60 maydetermine whether the differences between motion vectors in the Xdirection of blocks (e.g., macroblocks) in the candidate reference frameand motion vectors in the X direction of co-located blocks in a previousframe exceed a threshold. In this case, MV reliability checker 60 maydetermine that the motion vectors in the candidate reference frame arenot reliable. For example, MV reliability checker 60 may count thenumber of unreliable motion vectors in the candidate reference frame, ordetermine an overall average difference between motion vectors in thecandidate reference frame and the previous reference frame.

In addition to determination of motion vector reliability in the X(e.g., horizontal) direction, MV reliability checker 60 may determinemotion vector reliability in the Y (e.g., vertical) direction in asimilar manner. If MV reliability checker 60 detects either X directionMV unreliability or Y direction MV unreliability, the MV reliabilitychecker may reject the current candidate reference frame underconsideration and indicate to selection unit 44 that the candidatereference frame should not be selected as a reference frame forinterpolation of a skipped frame. As an alternative, in someimplementations, angle information may be used to evaluate both motionvector magnitude and direction. If MV's for the candidate referenceframe are reliable, however, MV reliability checker 60 indicates toselection unit 44 that the candidate frame may be selected as areference frame for interpolation or extrapolation of the extra frame.

As an alternative, MV reliability checker 60 may analyze MV reliabilityusing a frame-to-frame motion change detection technique. According tothis technique, MV reliability checker 60 may be configured to detectwhen the motion in the current candidate reference frame has changedsubstantially from the motion in another, adjacent frame, such as aprevious or future frame. If the magnitude of the change is greater thana threshold value, MV reliability checker 60 determines that the MVs ofthe candidate reference frame are unreliable and should not be selectedas a reference frame for frame substitution.

For the frame-to-frame motion change detection technique, to detectwhether motion in the candidate reference frame and the adjacent frameis continuous, the following two methods may be used. First, motionchange detection may be based on motion statistics. In this case, motionvector statistics may be calculated for both frames, i.e., the candidatereference frame and the adjacent frame. The statistics may includemotion vector (magnitude and angle) mean and standard deviation. Second,motion change detection may be based on motion vector labeling. In thiscase, motion change detection based on statistics may rely on the rangeof motion on the frame level to make decision. The difference of motionfor each co-located macroblock in the two frames may not be detected. Tosolve this problem, however, motion change detection based on motionvector labeling may be used.

As a further alternative, MV reliability checker 60 may analyze MVreliability using a motion trajectory based technique. In this case, themotion trajectory based approach determines whether a motion vector fromthe adjacent frame should be used by looking at where the macroblockwould be in the candidate frame if it had followed the same motiontrajectory as in the adjacent frame. If the object carried by themacroblock has significant overlap with the area of interest in thecandidate frame (i.e., the location of the lost macroblock), then its MVmay be considered reliable and the candidate frame should be used forframe substitution. Otherwise, if it moves away from the area ofinterest, then its MV is not reliable and the candidate frame should notbe used for frame substitution.

The quality threshold value used by compare unit 59 may be predeterminedand selected to represent a quality level that is believed to correlatewith acceptable frame substitution results. The quality threshold valuemay be fixed or adjustable. In some implementations, for example, thequality value threshold value may be adjustable as a function of a modein which decoder 14 operates. In operation, analysis unit 42 may firstcheck whether the resource-focused mode is on or off. If mode selectionunit 46 indicates the quality-focused mode, mode adjustment unit 62 maynot adjust the existing quality threshold value. If mode selection unit46 indicates the resource-focused mode, however, mode adjustment unit 62may adjust the quality threshold value. For example, in someimplementations, mode adjustment unit 62 may increase the qualitythreshold value to require the selection of higher quality frames asreference frames for interpolation or extrapolation of a frame.

In some implementations, when the resource-focused mode is selected, thequality threshold value may be increased from a first value to a second,increased value. In other implementations, when the resource-focusedmode is selected, the quality threshold value may be increased to avalue that is computed as a function of an available resource level. Asresources such as available power or available computing resourcesbecome lower, for example, the quality threshold value may be higher. Inthis manner, the quality threshold value may be inversely proportionalto the available resource level such that higher quality is required tojustify the cost of frame substitution, i.e., cost of frameinterpolation or extrapolation, when resource levels are lower. Hence,in some examples, in the resource-focused mode, analysis unit 42 mayapply a motion activity threshold to determine whether to enable ordisable frame substitution and, when frame substitution is enabled, useeither a fixed quality threshold value or a quality threshold value thatis adjusted as a function of available resource levels.

If none of the candidate reference frames for interpolation orextrapolation of a particular frame produce a quality score thatsatisfies the increased quality threshold value in the resource-focusedmode, then no frames are selected for interpolation or extrapolation ofthe frame. In this case, selection unit 44 disables interpolation, inwhich case frame substitution unit 38 may suspend frame substitution andinstead apply frame repetition. By requiring an increased qualitythreshold value in the resource-focused mode, mode adjustment unit 62,in effect, compels frame substitution unit 38 to disable framesubstitution unless higher quality reference frames are available forframe substitution.

Mode selection unit 46 may select the resource-focused mode whenresources are limited, e.g., when power, computing and/or memoryresources are scarcer. The resource-focused mode may disable video frameinterpolation when the benefit of video frame interpolation is not verynoticeable, such as in the case of very low motion video content. Inthis manner, mode selection unit 46 and mode adjustment unit 62 impose ahigher quality justification for frame interpolation or extrapolationwhen resources are limited. In other words, to justify the cost of framesubstitution in terms of resource consumption, the actual results of theframe substitution should be of relatively high visual quality. Whenresources are less scarce, such that mode selection unit 46 selects thequality-focused mode, mode adjustment unit 62 may reduce the qualitythreshold value. The reduced quality threshold value permits framesubstitution to proceed on a more frequent basis, as more candidateframes may be likely to satisfy the reduced quality threshold value.

In the resource-focused mode, mode selection unit 46 also may activatemotion analyzer 64. In addition to activating mode adjustment unit 62 insome implementations to adjust the quality threshold value, motionanalyzer 64 may analyze motion activity of one or more candidatereference frames to determine whether the pertinent video scene isrelatively static or not. Motion analyzer 64 may analyze motion vectordata from the candidate reference frames to determine whether the videoscene is characterized by very little motion or significant motion. Forexample, motion analyzer 64 may analyze motion vectors of a currentanchor frame to make a decision whether to enable or disable video framesubstitution. The anchor frame may be a frame that is adjacent to, e.g.,before or after, the skipped frame.

If there is very little motion indicated by the anchor frame, such thatthe scene is relatively static, motion analyzer 64 may generate anoutput to disable frame substitution. If the scene is relatively static,interpolation of a temporal, skipped frame ordinarily will not bejustifiable, relative to frame repetition, even if the selectedreference frames would produce a high quality frame substitution result.As an illustration, when the scene is relatively static, visual qualitydifferences produced by frame interpolation and frame repetition may berelatively imperceptible. For this reason, the cost of interpolation andpower consumed by video data traffic between video buffer and display isnot justified by any significant gains in quality, and should bedisabled. In this case, frame repetition may be more desirable from aquality and resource conservation perspective. When motion analyzer 64indicates that frame substitution should be disabled, thequality-focused analysis for the currently considered reference framemay likewise be disabled.

In some implementations, motion analyzer 64 may compare the motionactivity to a threshold value. The motion threshold value may be fixedor adjustable. For example, motion analyzer 64 may adjust the motionthreshold value based on a level of available power, computing and/ormemory resources. In the adjustable case, if the power resources are ata relatively high level, for example, the motion activity thresholdvalue may be relatively low. If the power resources are at a relativelylow level, for example, the motion threshold value may be relativelyhigh. In either case, motion activity at or above the threshold valuemay trigger interpolation, subject to selection of one or more referenceframes for use in frame substitution, consistent with thequality-focused mode of operation. For higher power levels, a lowerthreshold value means that less motion may be required to trigger framesubstitution. For lower power levels, however, a higher threshold valuemeans that higher motion may be required to trigger frame substitution.

Motion analyzer 64 and, optionally, mode adjustment unit 62, may beactivated when mode selection unit 46 selects the resource-focused modeand deactivated when mode selection unit 46 selects the quality-focusedmode. In the quality-focused mode, decoder 14 may operate to producedesirable visual quality. In the resource-focused mode, however, decoder14 may combine both quality and resource conservation objectives. Modeselection unit 46 may select the resource-focused mode in response todetection of limited resources, e.g., by comparison of availableresources to one or more resource thresholds. Hence, in someimplementations, mode selection unit 46 may select the quality-focusedmode by default and select the resource-focused mode based on the levelof available resources. Alternatively, the resource-based mode may bethe default mode, in which case mode selection unit 46 selects thequality-focused mode when available resources are above one or moreresource thresholds.

In effect, mode selection unit 46 may direct selection unit 44 toselectively enable or disable interpolation or extrapolation of theadditional video unit in the resource-focused mode, and direct theselection unit to enable interpolation or extrapolation of theadditional video unit in the quality-focused mode. In particular, modeselection unit 46 may direct selection unit 44 to selectively enable ordisable interpolation or extrapolation in the resource-focused mode bytriggering motion analyzer 64. Selection unit 44 may then selectivelyenable or disable interpolation or extrapolation based on the output ofmotion analyzer 64. Alternatively, selection unit 44 may directselection unit 44 to enable interpolation or extrapolation, e.g., by nottriggering motion analyzer 64. Enablement of interpolation by selectionunit 44 may still be subject to quality analysis by analysis unit 42 toselect reference frames or to disable interpolation or extrapolation ifno suitable reference frames are available.

As an additional, optional feature, in some aspects, analysis unit 42may include a distance unit 63. As described above with reference toFIG. 4, a delay detection unit 51 may detect the operation of adelay-sensitive video application, such as a video telephonyapplication. Delay detection unit 51 may direct selection unit 44 toavoid selection of future reference frames that are further in thefuture, relative to the frame to be added, in order to avoid delays inprocessing and presentation which could disrupt the visual quality ofthe video telephony. Accordingly, selection unit 44 may reject somecandidate reference frames based at least in part on analysis oftemporal distance of such frames from a frame to be added even thoughanalysis unit 42 may indicate a relatively high quality for such frames.Although selection of reference frames based on temporal distance may betriggered based on detection of a delay-sensitive application, in someimplementations, temporal distance may be used for reference frameselection on a regular basis, i.e., with or without detection of adelay-sensitive application.

As shown in FIG. 5, the delay feature may be built into analysis unit42. In particular, instead of rejecting, via selection unit 44, futurecandidate reference frames that reside too far in the future relative tothe frame to be added (i.e., based on excessive temporal distance),analysis unit 42 may be configured to assign relatively low qualityscores to such frames. Selection unit 44 may not reject such framesbased directly on temporal distance, but indirectly based on a lowquality score resulting from the temporal distance. In either case,selection unit 44 can avoid selection of temporally distant referenceframes that may introduce excessive latency into the frame substitutionprocess. Again, temporal distance may be used for selection of referenceframes on a selective basis when operation of a delay-sensitiveapplication such as video telephony is detected, or on a regular basis,e.g., whether a delay-sensitive application is detected or not. In somecases, reference frames may be selected based at least in part ontemporal distance on a regular basis without an operation that involvesdetection of a delay-sensitive application.

Distance unit 63 may determine, as one of the various framecharacteristics, a distance of a future candidate reference frame from aframe to be added. Distance unit 63 may be configured to generateprogressively lower scores for future candidate reference frames thatare further away from the frame to be added. In some cases, distanceunit 63 may produce a score of zero if a candidate reference frame ismore than a maximum number of frames in the future relative to the frameto be added.

Distance unit 63 may be activated by delay detection unit 51 when adelay-sensitive video application is detected. Alternatively, delaydetection unit 51 may cause the output of distance unit 63 to carry anincreased weighting when a delay-sensitive application is detected. Whenan application, such as video playback, is not significantlydelay-sensitive, distance unit 63 may be disabled or its output scoremay carry a reduced weighting in the overall quality score.

The various components, units or modules included in system 10, encoder12 and decoder 14 in FIGS. 1-5, as well as other components describedthroughout this disclosure, may be realized by any suitable combinationof hardware and/or software. In FIGS. 1-5, various components aredepicted as separate components, units or modules. However, all orseveral of the various components described with reference to FIG. 1-5may be integrated into combined units or modules within common hardwareand/or software. Accordingly, the representation of features ascomponents, units or modules is intended to highlight particularfunctional features for ease of illustration, and does not necessarilyrequire realization of such features by separate hardware or softwarecomponents. In some cases, various units may be implemented asprogrammable processes performed by one or more processors. For example,in various aspects, a motion analyzer, a resource monitor, and aselection unit could be realized by separate hardware and/or softwareunits, or the same hardware and/or software units, or combinationsthereof.

FIG. 6 is a flow diagram illustrating an exemplary technique in whichvideo decoder 14 selects reference frames for video frame substitution.The process shown in FIG. 6 may be performed in the quality-focusedmode, or in the resource-focused mode when frame substitution isenabled. As shown in FIG. 6, video decoder 14 receives input videoframes (68). In the case of decoder 14, the input video frames may bereceived as encoded frames in an incoming bitstream, where some of theframes are missing due to frame skipping or the basic frame rate ofencoder 12. In the case of an encoder 12, the input video frames may besource video frames to be encoded by the encoder. The temporal, excludedframes may be frames that have been intentionally skipped by videoencoder 12 or frames that are lost in transmission across channel 19, orframes that were not supported by the basic frame rate of encoder 12 andneed to be additionally created for frame rate conversion. In theexample of FIG. 6, analysis unit 42 analyzes one or more characteristicsof a set of candidate reference frames (70) for use in framesubstitution.

The candidate reference frames may be selected from N previous framesand M future frames relative to the frame to be added, where N and M maybe equal or unequal. As one illustration, three previous frames andthree future frames may be considered, although the particular number offrames is for purposes of illustration and should not be consideredlimiting. Previous and future frames may be considered for bidirectionalinterpolation. For unidirectional interpolation or extrapolation, insome aspects, only previous frames or only future frames may beconsidered.

The one or more characteristics of the frames may relate to quality ofthe frames, and may be analyzed using pixel domain values, transformdomain values, bitstream data, or otherwise. Based on the analysis,analysis unit 42 selects one or more of the candidate frames asreference frames (72). Using the selected candidate frames as referenceframes, frame substitution unit 38 performs frame substitution byinterpolation (or extrapolation, if applicable) of the frame to be added(74). The process outlined in FIG. 6 may be repeated on a substantiallycontinuous basis by decoder 14 for interpolation (or extrapolation) ofadditional video frames in the received bitstream.

FIG. 7 is a flow diagram illustrating an example technique for referenceframe selection in more detail. In general, analysis unit 42 maydetermine whether a resource-focused mode, such as a power-saving mode,is ON or OFF. If the resource-focused mode is ON, motion analyzer 64analyzes motion activity of the candidate reference frame. For example,the received motion vectors for a candidate reference frame areanalyzed. Based on motion vector analysis, a decision is made whether tointerpolate (or extrapolate) a frame to be added or not. If the decisionis to interpolate, the quality of reference frames are analyzed next,consistent with the quality-focused mode. Based on this analysis,another decision on whether to interpolate or extrapolate is made. Ifinterpolation or extrapolation is elected, a reference frame (or frames)is selected based on quality analysis. Video frame interpolation orextrapolation is then performed using at least the selected referenceframe and possibly multiple reference frames.

The process outlined in FIG. 7 may be repeated on a substantiallycontinuous basis for each frame to be added, or selected frames to beadded, in a video sequence received by video decoder 14. In the exampleof FIG. 7, decoder 14 receives an incoming bitstream containing encodedinput video frames of a video sequence and excluding some frames (76).For interpolation or extrapolation of a missing frames, video decoder 14may determine whether a resource-focused mode is ON or OFF (78). Forexample, mode selection unit 46 may indicate whether the framesubstitution process of decoder 14 should operate in a resource-focusedmode or a quality-focused mode.

If the resource-focused mode is activated (78), analysis unit 42 mayactivate motion analyzer 64 to analyze the motion indicated for each ofthe candidate reference frames (80). For example, motion analyzer 64 mayanalyze motion vector data from the candidate reference frames todetermine whether the video scene is relatively static or includessignificant movement. Motion analyzer 64 may analyze motion vectors ofan anchor frame, such as a frame that is temporally adjacent to asubstitute frame to be added, to make a decision whether to enable ordisable video frame substitution.

If the motion level indicated by the anchor frame is greater than orequal to a threshold motion level (82), motion analyzer 64 may indicateto selection unit 44 that the level of motion is sufficient to justifyframe substitution by interpolation (or extrapolation) of the frame tobe added. In this case, analysis unit 42 may proceed to analyze thequality of the candidate reference frame or frames (84). If the motionlevel is below the threshold motion level, however, motion analyzer 64indicates to selection unit 44 that frame substitution should not beused, in which case decoder 14 may repeat a reference frame (92) inplace of the excluded frame instead of interpolating the frame. In thiscase, a previous or future frame may be repeated to effectivelyupconvert the frame rate of the video sequence.

In some implementations, for frame repetition, analysis unit 42 maysimply select a previous frame or future frame and repeat that frame inplace of the frame to be added. Because frame repetition is selected,there is no need to perform the various quality analysis operations ofthe quality-focused mode for the current frame. As an alternative,however, analysis unit 42 may apply the quality analysis to select areference frame from one of a plurality of candidate reference framesand then use the selected reference frame as the repeated frame. Inparticular, analysis unit 42 may select a reference frame having a totalquality score that satisfies, e.g., meets or exceeds, the qualitythreshold and use the selected frame for frame repetition. Hence, insome cases, quality-based selection of reference frames consistent withthe quality-focused mode may be used not only for frame interpolation orextrapolation, but also for frame repetition.

If the motion level is at or above the threshold level (82), asdiscussed above, analysis unit 42 may proceed with the quality-focusedmode of operation to analyze the quality of the candidate referenceframe or frames and compare the resulting quality score to a qualitythreshold value (86). As described with reference to FIG. 5, forexample, quality score calculator 58 may calculate a total quality scorebased on quality scores generated by one or more of objective metricchecker 50, EC checker 52, QP checker 54, and CBP checker 56. Compareunit 59 may then compare the total quality score and compare it to thequality threshold value (86).

If the quality score is satisfactory, e.g., greater than or equal to thethreshold (86), analysis unit 42 may indicate selection of the referenceframe or frames to frame substitution unit 38. For example, selectionunit 44 may identify the reference frame or frames for selection byframe substitution unit 38 for use in interpolation (or extrapolation)of the frame to be added. Frame substitution unit 38 may then proceed toperform frame substitution, e.g., by interpolating or extrapolating theframe using the selected reference frame or frames (90). If none of thecandidate reference frames has a total quality score that issatisfactory, e.g., meets or exceeds the quality threshold value (86),compare unit 59 may indicate to selection unit 44 that no referenceframes should be selected. In this case, selection unit 44 may indicateto frame substitution unit 38 that frame substitution should be disabledand that frame repetition should be applied (92) instead of framesubstitution.

As shown in FIG. 7, analysis unit 42 may be optionally configured torank candidate reference frames having total quality scores that satisfythe quality threshold (93). For example, for bi-directionalinterpolation, analysis unit 42 may rank previous candidate referenceframes in order of quality level, rank future candidate reference framesin order of quality level, where previous frames are temporally prior tothe frame to be added and future frames are temporally after the frameto be added, and then select the most highly ranked previous frame andthe most highly ranked future frame for use in frame substitution. Inthe event of a substantial tie in the rankings, analysis unit 42 mayselect the frame that is temporally closest to the frame to be added.For unidirectional interpolation, analysis unit 42 may select the mosthighly ranked previous frame or the most highly ranked future frame,depending on whether the previous or future frame will be used as thereference frame. As a further example, in some cases, ranking may beused to select the most highly ranked frame for frame repetition, aswell as frame substitution.

In some implementations, selection of the frames may be subject to MVreliability analysis by MV reliability checker 60, as described withreference to FIG. 5. In addition, in some implementations, the qualitythreshold value and/or other criteria or operations may vary between thequality-focused mode and the resource-focused mode. For example, modeadjustment unit 62 may increase the quality threshold value in theresource-focused mode to require higher interpolation quality to justifyinterpolation when resources are limited.

FIG. 8 is a flow diagram illustrating an example technique for qualityanalysis of reference frames to support reference frame selection forvideo frame interpolation in accordance with a quality-focused mode. Ingeneral, the quality of candidate reference frames is estimated oranalyzed. A quality score may be given for each candidate referenceframe. The candidate reference frame may be a previous or future frame,relative to a skipped frame. If the quality score of a particularreference frame is not greater than or equal to a threshold, it is notselected to be used in video frame substitution. If the quality score ofthe reference frame is adequate, then the reference frame may beselected to be used in video frame interpolation or extrapolation. Insome implementations, analysis unit 42 may require that motion vectorsassociated with the candidate reference frame are reliable. For example,motion vectors may be checked if video frame substitution uses amotion-compensated prediction method for interpolation or extrapolation.

As shown in the example of FIG. 8, analysis unit 42 may analyze multiplecandidate reference frames in succession to identify reference frames,including previous frames and future frames, for interpolation orextrapolation of a frame. Upon retrieving the next candidate referenceframe (94) from received frame buffer 34, analysis unit 42 estimates thequality of the candidate reference frame (96) and calculates a qualityscore (98), e.g., as described above with reference to FIG. 5. Is thescore is satisfactory, e.g., greater than or equal to the qualitythreshold value (100), compare unit 59 passes the candidate referenceframe to MV reliability checker 60 to determine whether the MV contentof the frame is reliable (102).

If the total quality score is not greater than or equal to the qualitythreshold value (100), analysis unit 42 may set the candidate referenceframe to “OFF” for purposes of selection for frame substitution (104).In this case, selection unit 44 does not select the “OFF” frame forinterpolation or extrapolation of a substitute frame by framesubstitution unit 38. If the total quality score is greater than orequal to the quality threshold value (100), and Mv reliability checker60 determines that the MV content is reliable (106), analysis unit 42may set the candidate reference frame to “ON” for purposes of selectionfor frame substitution (106).

If analysis unit 42 has not considered all of the candidate referenceframes, i.e., it has not reached the end of the candidate referenceframes (108) in a predefined range of candidate reference frames,analysis unit 42 retrieves the next candidate reference frame foranalysis. The number of candidate reference frames analyzed by analysisunit 42 may be preselected, and may include one or more previous framesand one or more future frames relative to the frame to be interpolatedor extrapolated, e.g., as described with reference to FIGS. 2A-2D. Whenthe end of the candidate reference frames is reached (108), selectionunit 44 may select the candidate reference frames that have been set to“ON” (110) and communicate the selected frames to frame substitutionunit 38. Frame substitution unit 38 then may interpolate or extrapolatethe frame to be added too perform frame substitution using the selectedreference frame or frames (112).

If there are no reference frames set to “ON,” or an insufficient numberof “ON” frames, selection unit 44 may indicate that frame substitutionshould be disabled, and that frame substitution unit 38 should insteadapply frame repetition for the frame to be added. In someimplementations, the FRUC process in FRUC unit 22 may be enabled anddisabled on a selective basis. When FRUC is enabled, the processoutlined in FIG. 8 may be repeated on a substantially continuous basisfor each frame to be added, or selected frames, in the video sequencereceived by video decoder 14.

Again, as described with reference to FIG. 7, the process shown in FIG.8 may include a ranking operation in which candidate reference framesthat satisfy the quality threshold value are ranked. In this case, thehighest ranked frames may be selected as reference frames for framesubstitution.

FIG. 9 is a flow diagram illustrating an example technique forgenerating quality scores for reference frames to support referenceframe selection for video frame interpolation. The quality scores may beused to generate a total quality score, e.g., as described withreference to FIG. 5. In general, as an initial step, average QP valuesof each reference frame (future or previous reference frame relative toa skipped frame) as well as CBP values may be checked. If the QP valueis smaller (e.g., a smaller QP value in H.264 coding corresponds to afiner quantization step size) than a threshold value, a high qualityscore may be given to the reference frame. The inverse may be true forsome coding processes other than H.264 in which a smaller QP valuecorresponds to a coarser quantization step size.

In addition, a determination may be made as to whether a slice loss ormultiple slice loss is present in the candidate reference frame. Ifthere is a loss and if no error concealment is applied, then the qualityscore of the reference frame may be reduced. If there is a loss and theerror is concealed, a higher quality score may be set for the referenceframe. In some implementations, objective, no-reference visual qualitymetrics (such as blockiness, blurriness and/or color bleeding) may beapplied to the reconstructed candidate reference frames. If the metricsprovide high results, the total quality score of the reference may beincreased. If a reference frame has a high total quality score, it maybe used in interpolating the temporal skipped frame.

As shown in FIG. 9, analysis unit 42 retrieves the next candidatereference frame from received frame buffer 34 (114), and analyzes the QPand CBP values for the frame. In the example of FIG. 9, analysis unit 42generates a combined QP-based score based on the QP and CBP values. Ifthe QP value is less than an applicable QP threshold value (QP_th) andthe CBP value is not equal to zero (116), analysis unit 42 sets aQP-based score for the frame to “HIGH” (118). If either the QP value isgreater than or equal to the QP threshold value (QP_th) or the CBP valueis approximately equal to zero (116), analysis unit 42 sets the QP-basedscore for the frame to “LOW” (120).

As further shown in FIG. 9, analysis unit 42 also may be configured tocheck the candidate reference frame for slice loss (122). With referenceto FIG. 5, slice loss checking may be performed by EC checker 52. Aslice loss may result from loss across channel 19 or some other loss orcorruption of data. If there is a slice loss, analysis unit 42 maydetermine whether an adequate error concealment (EC) mechanism isavailable to correct for the slice loss (124). If not, analysis unit 42sets an EC-based score for the candidate reference frame to “LOW” (126).If an adequate EC mechanism is available (124), analysis unit 42 setsthe EC-based score to “HIGH” (128). In this case, if the lost slice canbe reproduced using the EC mechanism, the quality of the candidatereference frame may be suitable for use as a reference frame for framesubstitution. If the slice cannot be reproduced, however, the candidatereference frame should not be used for frame substitution.

Analysis unit 42 also may apply objective quality metrics to thecandidate reference frame (130). For example, analysis unit 42 mayanalyze the decoded and reconstructed version of the candidate referenceframe to analyze any of a variety of objective metrics such asblockiness, blurriness, color bleeding or the like. Hence, the objectivemetrics may be applied in the pixel domain to evaluate visual quality.Analysis unit 42 may quantify each of the objective metrics to formulatea quality metric score QM. If the quality metric score QM is greaterthan an applicable quality metric threshold (QM_th) (132), analysis unit42 sets a QM-based score for the candidate reference frame to “HIGH”(134). If the quality metric score QM is less than or equal to thequality metric threshold (QM_th), analysis unit 42 sets the QM-basedscore to “LOW” (136).

Analysis unit 42 may set a total quality score for the candidatereference frame (138) based on a combination of the QP-based score, theEC-based score and the QM-based score. The total quality score may becompared to a threshold to determine whether to select the candidateframe for use in frame substitution. As described with reference to FIG.5, the total quality score may be formulated according to a weightedsummation of the QP-based score, the EC-based score and the QM-basedscore, where analysis unit 42 may attribute different weights to each ofthe individual scores. The order in which the individual QP-based score,the EC-based score and the QM-based score are obtained may be differentfrom the order shown in FIG. 9. In addition, the particular types ofscores shown in FIG. 9 may be subject to variation. In general, analysisunit 42 may be configured to obtain quality scores that indicate alikelihood that the candidate reference frame will contribute to aninterpolation result with an acceptable level of quality.

FIGS. 10 and 11 are flow diagrams illustrating example techniques forselective frame substitution in a resource-focused mode. In aresource-focused mode, analysis unit 42 may use motion activity as ameasure to determine if a video scene represented by a reference frameis static or not. A zero motion vector count and/or small motion vectorcount may be used as decision criteria. In general, thresholds may bederived in two ways. In a non-adaptive case, for example, a fixedthreshold for zero motion vector count and a fixed threshold for smallmotion vector count may be used to judge motion activity. In anadaptive-case, one or both of the thresholds may be adjusted based on,for example, the resource levels of decoder 14, such as levels ofavailable power, computing resources, or memory.

In general, a resource-focused mode may be an optional mode of selectiveframe substitution in which interpolation or extrapolation is turnedoff, i.e., disabled, when there is no substantial perceptible differencebetween the results of frame substitution and frame repetition, e.g.,when a difference between interpolation and frame repetition is notsufficient to justify the use of interpolation, e.g., in view of power,computing or memory constraints within the device associated withdecoder 14. When motion activity is significant, however, decoder 14may, in effect, revert to the quality-focused mode to select referenceframes for frame substitution.

In some implementations, a resource-focused mode may be characterized asa power-save mode or power-optimized selective FRUC mode. In aresource-focused mode, motion activity may be used as a measure todecide if the video scene is static or not. If the algorithm determinesthat the scene is static, then a simple frame repetition technique maybe used for FRUC, instead of video frame substitution, which isgenerally more computationally intensive and consumes more power. If thevideo scene is not generally static, however, frame substitution may bemore desirable than frame repetition.

To determine whether a scene is static, motion vectors of a currentanchor frame may be analyzed to make a decision whether to enable ordisable video frame interpolation or extrapolation. The motion vectorsmay be used directly from the bitstream, obtained after processing ofbitstream motion vectors in decoder 14, or obtained from a motionestimation module of the decoder 14. In some cases, some of the motionvectors used for motion analysis may be the same motion vectors used forinterpolation or extrapolation. The anchor frame may be a frame adjacentto a frame to be added, such as a previous or future frame immediatelyadjacent or in proximity to the frame to be added. On the basis of thisanalysis, analysis unit 42 may make a decision on whether to enable ordisable video frame interpolation for the frame presently underconsideration. In one example, the number of zero motion vectors presentin the frame can be used as a decision criteria.

A zero motion vector is a motion vector with a value of zero, orsubstantially zero. For example, in some implementations, motion vectorswith values below a threshold value may be considered to be zero motionvectors. In some implementations, the motion vectors may be processedvalues from bitstream embedded motion vectors. If the zero motion vectorcount, i.e., the number of zero valued motion vectors, is greater than athreshold value, the scene may be determined to be static, in which casevideo frame interpolation is disabled. For static scenes, framerepetition may be used. Further enhancement may be added by using modedecision information, e.g., intra or inter coding mode decisioninformation. For example, the number of zero motion vectors may becounted for non-intra coded macroblocks to obtain a more precise zeromotion vector count.

In another example, a small motion vector count may be used as decisioncriteria in addition to zero motion vector count. A small motion vectormay be a nonzero motion vector with a value that is below apredetermined threshold. One reason for adding small motion vector countis that, for example, some scenes, even though they may have a largenumber of static macroblocks indicated by zero motion vectors, may alsocontain a relatively small number of fast moving objects such as athrown ball, a flying bird, or a passing vehicle. A new object thatquickly enters or leaves a video scene over a series of video frames mayproduce significant motion activity in the frames.

Although fast-moving objects may occupy a small portion of the overallframe, it may be important to interpolate them in the FRUC-generatedframe to preserve temporal quality. Hence, a second criterion thatdepends on small motion vector count, e.g., a small motion vector countthreshold, may be added to ensure that scenes are absolutely static whenframe interpolation is disabled. Furthermore, this criterion may accountfor the fast-moving small object problem implicitly.

In the example of FIG. 10, motion analyzer 64 counts the number of zerovalue motion vectors in the frame to produce the zero motion vectorcount (Zmv_c) (140). In general, each macroblock in the frame has amotion vector. If the motion vector for a macroblock has a value ofzero, indicating no movement, then the motion vector is counted as azero motion vector. Again, in some cases, a motion vector may be countedas a zero value motion vector if it has a small nonzero value below athreshold. In either case, once the number of zero motion vectors iscounted, motion analyzer 64 determines whether the number of zero motionvectors, i.e., the zero MV count Zmv_c, is greater than or equal to anapplicable threshold Th (142) If so, the video scene is relativelystatic. In this case, motion analyzer 64 communicates to selection unit44 that interpolation or extrapolation should be disabled and that framesubstitution unit 38 should instead apply frame repetition for the frameto be added (144). As mentioned previously, frame repetition may use asimple scheme of selecting the closest previous or future frame, or usea quality-based scheme to select a higher quality reference frame as therepeated frame.

If the zero MV count Zmv_c is less than the threshold (142), motionanalyzer 64 communicates to selection unit 44 that frame substitutionshould be performed (146). In this case, interpolation (orextrapolation) may or may not be performed, subject to the selection ofone or more suitable reference frames as a result of the qualityanalysis performed by analysis unit 42. In effect, when the scene is notstatic, analysis unit 42 may revert to the quality-focused mode toselect one or more reference frames, if frames having sufficient qualityfor interpolation are available. If the quality analysis produces one ormore suitable reference frames, selection unit 44 may communicate theselected frame(s) to frame substitution unit 38 for use in interpolationof the frame to be added.

The threshold Th (142) may be fixed. Alternatively, the threshold Th maybe adjusted based on available frame substitution resources of the videodecoder 14, such as a power level, a computing resource level, and/or amemory resource level. As shown in FIG. 10, in the resource-focusedmode, analysis unit 42 may optionally detect a resource level (143) andadjust the threshold Th based on the detected resource level (145). Forexample, analysis unit 42 may determine available power resources, suchas battery level, and/or available computing resources, number ofavailable instructions per second, and/or available memory resources.The resource levels may be detected directly or estimated, e.g., basedon known relationships between computing operations and resourceconsumption. In the case of a mobile device, if battery resources arelow, analysis unit 42 may reduce threshold Th so that frame substitutionis enabled when substantial amounts of motion are present. In eithercase, whether thresholds are fixed or adjustable, if frame substitutionis enabled, analysis unit 42 may apply one or more quality criteria toselect reference frames or disable frame substitution.

In the example of FIG. 11, motion analyzer 64 analyzes motion based onboth the zero Mv count and a small MV count. As shown in FIG. 11, motionanalyzer 64 counts the number of zero valued motion vectors in the frameto produce the zero motion vector count (Zmv_c) (148) and counts thenumber of small motion vectors in the frame to produce the small motionvector count (Smv_c) (150). Small motion vectors may be motion vectorswith nonzero values that are below a threshold value. Even when there isa predominance of zero motion vectors, there may be a number of nonzeromotion vectors, including nonzero motion vectors that are small in thesense that they have values below a threshold value (referred to hereinas small motion vectors). Small motion vectors may be associated withone or more small moving objects, such as a ball, bird, car, or thelike. Motion analyzer 64 compares the zero MV count (Zmv_c) and thesmall MV count (Smv_c) to respective thresholds Th1 and Th2,respectively (152).

If the zero MV count (Zmv_c) is greater than or equal to the thresholdTh1 and the small MV count (Smv_c) is less than the threshold Th2,motion analyzer 64 directs selection unit 44 to indicate that framerepetition should be applied instead of frame substitution (154). Inthis case, a zero MV count in excess of the threshold Th1 indicates thatthe video scene presented by the frame is generally static. At the sametime, a small MV count that is less than the threshold Th2 indicatesthat there are no significant small objects moving within the videoscene presented by the frame. In view of the generally static content ofthe frame, frame repetition is appropriate.

If the zero MV count (Zmv_c) is less than the threshold Th1 or the smallMV count (Smv_c) is greater than or equal to the threshold Th2, motionanalyzer 64 indicates that frame substitution should be performed (156),subject to selection of reference frames as part of the quality analysisperformed by analysis unit 42. In this case, a zero MV count that isless than the threshold Th1 indicates that the video scene includessignificant motion.

Even if the zero MV count is not less than the threshold Th1, the framemay include one or more relatively small moving objects that would bebetter presented by interpolation. Accordingly, motion analyzer 64 mayindicate that interpolation should be performed when the small MV countis greater than or equal to the threshold Th2, indicating the presenceof one or more small, fast-moving objects. Such objects generally may besmall and faster moving than other objects in a frame. When framecontent is not generally static or a generally static frame includes asmall moving object, frame substitution, e.g., by interpolation orextrapolation of the frame, may be appropriate.

As in the example of FIG. 10, the thresholds Th1, Th2 in FIG. 11 may befixed or adjusted based on available interpolation resources. Inparticular, one or both of the thresholds Th1, Th2 may be adjusted basedon a determined resource level. As shown in FIG. 11, in theresource-focused mode, analysis unit 42 may optionally detect a resourcelevel (143) and adjust the threshold based on the detected resourcelevel (145). Again, analysis unit 42 may detect or estimate availablepower resources, such as battery level in a mobile device, and/oravailable computing resources, number of available instructions persecond, and/or available memory resources. If battery resources are low,for example, analysis unit 42 may reduce threshold Th1 and increasethreshold Th2 so that interpolation is enabled when substantial amountsof motion are present.

In some cases, a threshold used to classify a motion vector as a smallmotion vector may be fixed or adjustable based on format size. Asdiscussed above, a small motion vector may be a nonzero motion vectorthat has a value below a particular threshold. In some implementations,the threshold used to determine whether a nonzero motion vector is smallor not may be adjusted based on a format size of the video units beingdecoded and interpolated or extrapolated. For example, a small motionvector may be classified by different thresholds for QCIF, CIF, QVGA andVGA frames, which having progressively larger format sizes. The value ofthe small motion vector threshold for a CIF frame may be smaller thanthe value of the small motion vector threshold for a VGA frame. Inparticular, a motion vector magnitude may be considered large in asmaller format frame but small in a larger format frame in considerationof the larger overall size of the larger format frame. Hence, in someimplementations, motion analyzer 64 may adjust the small motion vectorthreshold value based on the format size of the video unit that is beinginterpolated or extrapolated. For example, the small motion vectorthreshold value used for smaller format frames may be less than thesmall motion vector threshold value used for larger format frames.

FIG. 12 is a block diagram illustrating an example of a video decoder158 configured to selectively enable or disable display of substituteframes, e.g., based on an analysis of one or more qualitycharacteristics. For example, as will be described, video decoder 158may be configured to analyze one or more characteristics associated withan interpolated or extrapolated video frame produced by a frame rateupconversion process in a video decoder, and selectively enable anddisable presentation of the interpolated or extrapolated video unit on adisplay based on the analysis.

The analysis may involve analysis of any of a wide variety of qualitycharacteristics. The quality characteristics may include at least one ofa pixel domain characteristic, a transform domain characteristic, and/ormotion vector reliability. The one or more quality characteristics maybe used to formulate quality metrics including spatial quality metrics,temporal quality metrics, or other metrics. Such quality metrics may beuseful in predicting the impact of an interpolated or extrapolated frameon the visual spatial and/or temporal quality of the video sequence whenpresented to a user by a display. Decoder 158 may be configured toselectively disable display of a substitute frame, even though thesubstitute frame has already been interpolated or extrapolated, if aquality level associated with the substitute frame does not satisfy aquality threshold.

A quality level that is associated with a substitute frame may be basedon a quality level of the substitute frame itself. In someimplementations, the quality level may be based on a quality level ofone or more reference frames used to interpolate or extrapolate thesubstitute frame. In other implementations, the quality level may bebased on a quality level of the substitute frame and one or morereference frames using to produce the substitute frame. In each case,the quality level may generally indicate the degree of enhancement ofvisual quality that may be achieved by displaying the substitute frame.

Even when a substitute frame has already been interpolated orextrapolated, it still may be desirable to selectively disabletransmission and display of the substitute frame in a video decoder. Forexample, although interpolation or extrapolation has already beenperformed, the visual quality produced by displaying the substituteframe may be insufficient to justify expenditure of additional resourcesfor transmission and display of the substitute frame. Instead, it may bemore desirable to discard the substitute frame and repeat an adjacentframe, e.g., by holding the frame in the display for a longer period oftime instead of transmitting a new frame, thereby conserving power orother resources that would be necessary to display the substitute frame.In some implementations, in addition to selectively disablingtransmission and display of some frames, video decoder 158 also maydisable one or more post-processing operations such as smoothing,sharpening, brightness control, and/or contrast enhancement forsubstitute frames to conserve additional resources.

In the example of FIG. 12, video decoder 158 includes a received framebuffer 160 that receives encoded frames 24, e.g., from encoder 12, adecoding unit 162 that decodes the received frames, an output framebuffer 164 that stores the decoded frames, a substitution unit 166 thatperforms frame substitution by interpolation or extrapolation to addframes to output frame buffer 164, and thereby support a FRUC process,and a selective display analysis unit 168. Analysis unit 168 may beconfigured to generate a signal or command. In response to the signal orcommand from analysis unit 168, control unit 172 may selectively enableor disable transmission of substitute frames from a video buffer, suchas output frame buffer 164, to a display 170 for visual presentation toa user. In some aspects, analysis unit 168 may be selectively activatedwhen decoder 158 is in a resource-focused mode, e.g., as described withreference to FIGS. 4, 5 and 7. Alternatively, in other implementations,the selective display analysis unit 168 may operate on a regular basisto selectively enable or disable display of substitute frames producedby substitution unit 166 with or without selective activation.

Substitution unit 166 performs frame substitution, such as frameinterpolation or extrapolation, to add frames to the output framesstored in output frame buffer 164 and thereby support a FRUC process.Video decoder 158 may or may not apply reference frame selectiontechniques, as described elsewhere in this disclosure, to selectparticular reference frames for use by substitution unit 166 ininterpolation or extrapolation of a substitute frame. In some aspects,substitution unit 166 may select one or more reference frames based onan analysis of those frames, or simply use one or more adjacent frames,to produce the substitute frames. In either case, however, analysis unit168 may be further configured to analyze a quality level associated witha substitute frame to determine whether to display the substitute frame.The quality associated with the substitute frame may include one or morequality characteristics associated with the substitute frame itself, oneor more quality characteristics associated with one or more of thereference frames used to interpolate or extrapolate the substituteframe, or a combination of both. To analyze quality, analysis unit 168may apply objective quality metrics to the substitute (e.g.,interpolated or extrapolated) frame and/or the reference frames used tointerpolate or extrapolate the substitute frame.

Analysis unit 168 may direct control unit 172 to selectively enable ordisable the display of a substitute frame from output frame buffer 164via display 170 based on the quality analysis. For example, if thequality of one or more reference frames used to generate a substituteframe, and/or the quality of the substitute frame itself, do not satisfyan applicable quality threshold, analysis unit 168 may direct controlunit 172 to disable transmission of the interpolated frame from outputframe buffer 164 for presentation on display 170. If the quality isacceptable, however, the selective display analysis unit 168 may directcontrol unit 172 to enable transmission of the interpolated orextrapolated frame to display 170 for presentation to a user via display170. In this manner, video decoder 158 may avoid expenditure ofadditional resources to display a substitute frame when the quality ofthe substitute frame is unsatisfactory.

When a substitute frame is not sent to display 170, it may be discardedfrom output frame buffer 164, e.g., by overwriting the frame with aframe that is subsequently decoded, interpolated or extrapolated. Inthis case, display 170 may simply repeat the previous frame (e.g., byholding the previous frame in the display for a longer period of time)that was sent from the output frame buffer 164 to the display, ortransmit and repeat (e.g., hold) the next frame that follows thesubstitute frame. In this case, the display 170 may display the previousframe for an additional period of time, rather than display a substituteframe during that time. By using frame repetition, video decoder 159 mayavoid the need to transmit and display the substitute frame, therebyconserving power and/or other resources. As mentioned above, selectivedisplay of substitute frames may be used alone or in combination withreference frame selection techniques as described elsewhere in thisdisclosure.

Analysis unit 168 may be configured to analyze one or more of a varietyof quality characteristics associated with substitute frames produced bysubstitution unit 166. Quality characteristics analyzed by analysis unit168, in some implementations, may be similar to those evaluated byobjective metric checker 50 of FIG. 5. For example, analysis unit 168may be configured to analyze pixel values associated with substitute(e.g., interpolated or extrapolated) frames produced by substitutionunit 166, or pixel values associated with one or more reference framesused to produce the substitute frames, to determine degrees ofblockiness, blurriness, color bleeding or other objective spatialquality metrics associated with such frames, and generate a qualityscore based on the determination.

Hence, the quality analysis may be performed by analyzing the substituteframes produced by substitution unit 166 in the pixel domain, or byanalyzing decoded, reconstructed reference frames produced by decodingunit 162 in the pixel domain, or a combination of both. In animplementation in which reference frames are analyzed, the referenceframes may be the reference frames used to interpolate or extrapolatethe substitute frame. The quality score for a particular interpolated orextrapolated, substitute frame produced by substitution unit 166 may below when substantial blockiness, blurriness, and/or color bleeding isdetected in the substitute frame (or in one or more reference framesused to produce the frame) and high when there is substantially noblockiness, blurriness, and/or color bleeding.

The quality score for different substitute frames may vary between highand low as a function of such objective visual quality metriccharacteristics. Alternatively, the quality score may be expressed aseither high or low based on comparison to a predetermined threshold. Ineither case, if the quality score for a substitute frame does notsatisfy (e.g., is less than) a quality threshold, analysis unit 168 maydirect control unit 172 to disable transmission of the frame from outputframe buffer 164 to display 170. Alternatively, if the quality score forthe substitute frame satisfies (e.g., is greater than or equal to) thethreshold, analysis unit 168 may direct control unit 172 to enabletransmission of the substitute frame from output frame buffer 164 todisplay 170 for presentation to a user.

As described above, analysis unit 168 may analyze qualitycharacteristics associated with a substitute frame by analyzing pixelvalues of a substitute frame, pixel values of one or more referenceframes used to produce the substitute frame by interpolation orextrapolation, or pixel values of one or more other frames in thevicinity of the substitute frame. If reference frames are analyzed,substitution unit 166 may indicate to analysis unit 168 which referenceframe or frames were used to interpolate or extrapolate a particularsubstitute frame.

Using pixel values, as described above, analysis unit 168 may analyzeone or more spatial quality metrics such as SSIM, blockiness, blurrinessand/or color bleeding metrics associated with a substitute frame, and/orone or more reference frames used to produce the substitute frame. Insome implementations, a pixel value analysis alternatively, oradditionally, may be applied to other frames (in addition or as analternative to the reference frames or substitute frame) that aretemporally proximate to the substitute frame. If the spatial qualitymetric for a substitute frame, a reference frame, or another nearbyframe does not satisfy an applicable threshold, analysis unit 168 maydirect control unit 172 to disable display of the substitute frame.

As a simple illustration, if the amount of blockiness presented by pixelvalues associated with the substitute frame exceeds a threshold,analysis unit 168 may disable display of the substitute frame. In thismanner, decoder 158 may avoid displaying a substitute frame that couldadversely affect quality, or a substitute frame that provides a qualityenhancement that is insufficient to justify the expenditure ofadditional display resources.

As an alternative, or an additional operation, for quality analysis,analysis unit 168 may analyze one or more temporal quality metrics, suchas temporal fluctuation of spatial quality metrics. For example,analysis unit 168 may analyze fluctuation of spatial quality (e.g.,SSIM, blockiness, blurriness, and/or color bleeding) between asubstitute frame and one or more reference frames used to interpolate orextrapolate the substitute frame. If the temporal quality fluctuation isgreater than a fluctuation threshold, analysis unit 168 may directcontrol unit 172 to disable display of the substitute frame.

Analysis unit 168 may consider temporal fluctuation of one or morespatial quality metrics alone or in combination with a spatial qualitymetric itself. For example, analysis unit 168 may disable display of thesubstitute frame if either the spatial quality metric or the temporalquality metric does not satisfy an applicable threshold. Alternatively,analysis unit 168 may compute a score based on a weighted summation ofspatial quality and temporal quality values and compare the sum to acomposite threshold value.

As further refinements, in some implementations, analysis unit 168 maybe configured to analyze the locations of artifacts within a substituteframe, reference frame or other nearby frame. An artifact may be anundesirable visible feature that may be produced by blockiness,blurriness or color bleeding in a particular localized region of adisplayed frame. Analysis unit 168 may analyze overall blockiness,blurriness or color bleeding for a frame, or consider suchcharacteristics in the context of localized regions of a frame. Forexample, analysis unit 168 may analyze the pixel value variance oflocalized regions of a frame, e.g., 3 by 3 pixel regions, to generate anindication of texture in such regions. Smooth regions, generallyindicated by low variance, may be more susceptible to visible artifactscaused by blockiness, blurriness, color bleeding or other artifacts.However, such artifacts may be less visible in higher variance regionswith more texture.

Analysis unit 168 may produce localized spatial quality metrics formultiple localized regions within a frame, as an alternative or inaddition to a spatial quality metric for a frame. If the localizedspatial quality metrics for a frame do not satisfy an applicablethreshold in any of the smooth regions, where artifacts would be morevisible, analysis unit 168 may disable display of a substitute frame toreduce or avoid presentation of visible artifacts. If the localizedspatial quality metrics do not satisfy the quality threshold only inhigher variance regions with more texture, however, analysis unit 168may permit display of the substitute frame with the recognition thatartifacts will be invisible or less visible to the user in such regions.As a further alternative, analysis unit 168 may use different thresholdvalues for different localized regions. In a smooth, low varianceregion, analysis unit 168 may use a higher quality threshold, in effect,requiring higher quality to reduce or avoid introduction of visibleartifacts. In a higher variance region, analysis unit 168 may use alower quality threshold, permitting display of the substitute frame whensignificant artifacts occur only, or substantially only, in high textureregions.

Analysis unit 168 may scan the frame to analyze localized regions asdescribed above. If any of the localized regions, or a predeterminedpercentage of the localized regions, exhibits both a low variance (belowa variance threshold) and a low quality metric (below a qualitythreshold), which may indicate a likelihood of visible artifacts in thedisplayed frame, analysis unit 168 may direct control unit 172 todisable display of the substitute frame. The predetermined percentagemay be a threshold value that is fixed or adaptive. In otherimplementations, analysis unit 168 may be configured to analyze the sizeof regions in which artifacts may be visible, e.g., by consideringwhether localized regions having both low texture and low quality arecontiguous. Hence, analysis unit 168 may analyze a percentage of alllocalized regions having significant artifacts relative to a percentagethreshold and/or sizes of any of the artifacts (e.g., indicated by lowquality/low texture regions) relative to a size threshold to determinewhether to enable or disable display of a substitute frame.

Analysis unit 168 may additionally, or alternatively, analyze qualityassociated with the substitute frame using other types of data, such asdata indicating reliability of motion vectors associated with referenceframes used to interpolate or extrapolate the substitute frame. Otherexamples of data that may be used by analysis unit 168 for qualityanalysis include compressed domain information such as informationrelating to discrete cosine transform (DCT) or wavelet transformcoefficients associated with one or more reference frames used forinterpolation or extrapolation of a substitute frame. Compressed domaininformation may include, for example, QP and CBP values. As shown inFIG. 12, in some implementations, analysis unit 168 may receive QP andCBP values from decoding unit 162. The QP and/or CBP values may beassociated with reference frames used to interpolate or extrapolate asubstitute frame. Using QP, CBP values, and/or other compressed domaininformation, analysis unit 168 may analyze quality of substitute frames.

As an example, analysis unit 168 may evaluate motion vector reliabilityin a manner similar to that described above, e.g., with reference toFIGS. 5 and 8. If motion vector reliability of one or more referenceframes is unsatisfactory, analysis unit 168 may direct control unit 172to disable display of the substitute frame. Analysis unit 168 may applyany of a variety of techniques for determining motion vectorreliability, such as a motion difference based approach, aframe-to-frame motion change detection approach, or a motion trajectorybased approach, e.g., in a manner similar to that described above withrespect to MV reliability checker 60 of FIG. 5. Analysis unit 168 mayconsider motion vector reliability alone or in combination with otherquality metrics described herein, e.g., including pixel domain qualitymetrics, temporal metrics, or localized metrics, to determine whether toenable or disable display of a substitute frame.

As described above, analysis unit 168 may analyze quality based oncompressed domain information such as QP and/or CBP values associatedwith reference frames used to interpolate or extrapolate a substituteframe. The compressed domain information may be obtained from thereference frames in received frame buffer 160, e.g., by parsing thebitstream associated with the received frames. As an example, analysisunit 168 may be configured to analyze QP and/or CBP associated withreference frames in a manner similar to that described with reference toQP checker 54 and CBP checker 56, and quality score calculator 58 ofFIG. 5, and the process illustrated in the flow diagram of FIG. 9. Ifthe QP and CBP values indicate a relatively high quality for thereference frame, e.g., by comparing QP to a QP threshold QP_th anddetermining whether the CBP value is nonzero as shown in FIG. 9, thenanalysis unit 168 may direct control unit 172 to enable display of thesubstitute frame that was interpolated or extrapolated using thepertinent reference frame. If the QP and CBP values indicate a lowquality, however, analysis unit 168 may direct control unit 172 todisable display of the substitute frame.

The analysis of compressed domain information and MV reliability may beused by analysis unit 168 either alone or in combination with otherquality information, such as spatial quality information or temporalquality information, to determine whether to display a substitute frame.In some implementations, analysis unit 168 may be configured to analyzea variety of quality information in combination. For example, analysisunit 168 may analyze objective spatial and/or temporal quality metricsin the pixel domain (e.g., SSIM, blockiness, blurriness, and colorbleeding and/or associated temporal fluctuation), as well as transformdomain information such as QP and CBP values, and possibly errorconcealment (EC) reliability information, to generate a quality score,and then apply a motion vector (Mv) reliability analysis to determinewhether to display a substitute frame for which the quality score isacceptable.

For error concealment (EC) reliability, analysis unit 168 may apply anEC analysis similar to that described with reference to EC checker ofFIG. 5 and the processor of FIG. 9. For example, analysis unit 168 mayanalyze one or more reference frames used to produce a substitute frameand determine if there is a slice loss. If so, analysis unit 168 maydetermine whether a suitable EC mechanism is available. If there is nosuitable EC mechanism, analysis unit 168 may direct control unit 172 todisable display of the substitute frame by display 170. If an acceptableEC mechanism is available, however, analysis unit 168 may permit displayof the substitute frame. Analysis of slice loss and the EC mechanism maybe performed alone as a basis for selectively enabling or disablingdisplay of a substitute frame, or in combination with analysis of otherquality characteristics, such as spatial quality, temporal quality,motion vector reliability, or the like.

In some implementations, analysis unit 168 may be configured to operatein a manner similar to analysis unit 42 of FIG. 5, except that analysisunit 168 determines whether to display a substitute frame instead ofdetermining whether to select a particular reference frame of frames foruse in interpolating or extrapolating the substitute frame. Notably,because the substitute frame has already been produced by substitutionunit 166, analysis unit 168 may analyze objective qualitycharacteristics of the substitute frame in the pixel domain, either inaddition or as an alternative to analyzing quality characteristics ofone or more reference frames. For example, analysis unit 168 mayconsider the quality of the substitute frame alone or in combinationwith the quality and/or motion vector reliability of one or morereference frames. Alternatively, analysis unit 168 may consider thequality and/or motion vector reliability of the reference frames withoutconsidering the quality of the substitute frame.

Analysis unit 168 may be effective in selectively providing substituteframes to display 170 when such frames are more likely to contributefavorably to visual and/or temporal quality. Even though interpolationor extrapolation has already been performed, discarding the frame stillmay be advantageous if the quality level does not justify the additionalresources required to transmit and display the frame. Large amounts ofpower may be required to transmit a substitute frame from output framebuffer 164 to display 170.

In general, analysis unit 168 may be configured to analyze one or morequality and/or motion characteristics of an interpolated or extrapolatedvideo frame produced by a FRUC process performed by substitution unit166 in video decoder 158. Based on the analysis, analysis unit 168 maydirect control unit 172 to selectively enable and disable transmissionof the interpolated or extrapolated video frame for presentation ondisplay device 170. Although objective visual quality metriccharacteristics such as SSIM, blockiness, blurriness, and/or colorbleeding may be analyzed for quality analysis, other quality metrics maybe used.

In some aspects, decoder 158 may be configured to operate in differentmodes. In a first mode, such as a resource-focused mode, decoder 158 mayselectively enable and disable transmission of interpolated orextrapolated video frames for display on display device 170 based on thequality analysis, e.g., via control unit 172. In a second operatingmode, control unit 172 of decoder 158 may enable transmission of theinterpolated or extrapolated frame for presentation on display device170 without performing a quality analysis or without regard to theresults of a quality analysis.

Analysis unit 168 also may be configured to consider the number ofsubstitute frames for which display has been disabled over a past seriesof N previous substitute frames, where N represents a number greaterthan one. If more than a predetermined number or percentage M ofprevious substitute frames in the last N frames have not been displayed,e.g., due to quality characteristics, analysis unit 168 may enable thedisplay of the current substitute frame. In this case, the currentsubstitute frame may be displayed even though it may not satisfy theapplicable quality thresholds. In this manner, analysis unit 168 mayavoid skipping the display of a long series of substitute frames.

Alternatively, instead of enabling display, analysis unit 168 may reduceone or more applicable quality thresholds for the current substituteframe so that the substitute frame has a higher probability of beingdisplayed. Hence, in either case, analysis unit 168 may determinewhether to enable display of the current substitute frame based at leastin part on the number of substitute frames that have not been displayedover a predetermined number of previous frames. As an illustration, ifthe number of previous substitute frames under consideration is N=10 andthe threshold number of non-displayed frames M is 5, analysis unit 168may direct control unit 172 to permit display of the current substituteframe if the number of non-displayed frames in the last 10 frames was 5or more.

As a refinement, in some implementations, rather than considering theabsolute number of substitute frames that have not been displayed in thelast N substitute frames, analysis unit 168 may consider the number ofconsecutive substitute frames that have not been displayed. For example,analysis unit 168 may apply a consecutive count threshold of M. If theprevious M+1, consecutive substitute frames have not been displayed,e.g., due to quality characteristics, analysis unit may direct controlunit 172 to enable display of the current substitute frame. Analysisunit 168 may apply any of a wide variety of threshold schemes todetermine whether a current substitute frame should be displayed basedon the non-display of previous substitute frames. Accordingly, thisdescription is provided for purposes of illustration and withoutlimitation.

FIG. 13 is a flow diagram illustrating an example technique forselective display of substitute frames. Decoder 158 of FIG. 12 may beconfigured to perform the technique of FIG. 13. As shown in FIG. 13,decoder 158 may receive input video frames (174), decode the frames(176), and perform frame substitution (178) to produce substitute framesusing some of the received frames as reference frames for interpolationor extrapolation. An analysis unit, such as an analysis unit 168provided in decoder 158, may analyze one or more quality characteristicsof all or some of the substitute frames (180). In some implementations,the analysis unit may be provided in video decoder 158, a videopost-processing unit, or in a video display processing unit, such as amobile display processor (MDP). For example, analysis unit 168 mayanalyze various objective quality characteristics such as SSIM,blockiness, blurriness or color bleeding in an interpolated orextrapolated frame produced by substitution unit 166.

If the quality of the substitute frame satisfies a quality threshold,e.g., is greater than or equal to the quality threshold (182), analysisunit 168 may direct control unit 172 to enable display of the substituteframe (184), e.g., by enabling transmission of the frame from the outputframe buffer 164 to display 170. If the quality does not satisfy thequality threshold (182), e.g., is less than or equal to the threshold,analysis unit 168 may direct control unit 172 to disable display of thesubstitute frame (186). In some cases, the quality threshold may beadaptively adjusted, e.g., based on available resources. Hence, even ifsome resources have been expended to produce the substitute frame,decoder 158 may avoid consumption of additional resources when qualityof the substitute frame is not satisfactory, e.g., in relation to apredefined threshold quality level.

In the example of FIG. 13, analysis unit 168 may be configured toanalyze the quality of a substitute frame itself to determine whether todisplay that substitute frame. Hence, in some cases, there may be noneed to analyze other frames. In other implementations, however,analysis unit 168 may analyze the quality of one or more referenceframes used to produce a substitute frame as the basis for determiningwhether to display the substitute frame. As indicated by block 188,analysis unit 168 may optionally analyze the quality of one or morereference frames either alone or in combination with analysis of thequality of the substitute frame. The quality analysis of the substituteframe may generally involve analysis of pixel domain values of thesubstitute frame. As described above, quality analysis of the referenceframes may involve analysis of pixel domain values, transform domainvalues, MV reliability, EC reliability, or the like.

FIG. 14A is a block diagram illustrating an analysis unit 168 that maybe used with a video decoder 158 as shown in FIG. 12. In the example ofFIG. 14A, analysis unit 168 is configured to analyze quality associatedwith a substitute frame using objective one or more quality metrics forthe substitute frame. For example, an objective metric checker 173 mayanalyze objective spatial quality metrics such as SSIM, blockiness,blurriness, or color bleeding for the substitute frame. Such metrics maybe obtained, for example, from pixel values of the substitute frame. Insome implementations, objective metric checker 173 may analyze frames asa whole or on a localized region basis. Quality score calculator may 175may generate a quality score based on the objective quality metric ormetrics. Compare unit 177 may compare the quality score to a thresholdto determine whether to enable or disable display of the substituteframe. If the quality score satisfies the quality threshold, compareunit 177 may direct control unit 172 to permit display of the substituteframe via display 170. If the quality score does not satisfy the qualitythreshold, compare unit 177 may direct control unit 172 to disabledisplay of the substitute frame. In this case, a previous or futureframe may be repeated by display 170 instead of the substitute frame.

FIG. 14B is a block diagram illustrating another analysis unit 168 thatmay be used with a video decoder 158 as shown in FIG. 12. In the exampleof FIG. 14A, analysis unit 168 is configured to directly analyze thequality of the substitute frame. In the example of FIG. 14B, analysisunit 168 may be configured to analyze the quality of reference frames,the quality of the substitute frame, or a combination of both. Inaddition, in FIG. 14B, analysis unit 168 may be configured to analyzepixel values, transform domain values, motion vectors, and/or othertypes of information. In general, analysis unit 168 may be configured ina manner somewhat similar to analysis unit 42 of FIG. 5. For example,analysis unit 168 may include one or more of an objective metric checker173, quality score calculator 175, compare unit 177, EC checker 179, aQP checker 181, a CBP checker 183, and an MV reliability checker 185.

Objective metric checker 173 may analyze a substitute frame and/or oneor more reference frames used to produce the substitute frame to producean indication of objective spatial quality and/or temporal quality, asdescribed above. In some implementations, objective metric checker 173may analyze frames as a whole or on a localized region basis. EC checker179 may detect slice loss and determine whether an acceptable ECmechanism is available. QP checker and CBP checker 181, 183 may generatean indication of quality based on QP and CBP values for referenceframes.

Quality score calculator 175, in some implementations, may calculate anoverall quality score for consideration in selective display of asubstitute frame, e.g., in a manner similar to that described withreference to FIGS. 5 and 9. The quality score may be reset for eachsubstitute frame. However, if quality metrics that have been analyzedfor a previous substitute frame are useful for the current substituteframe, they may be retained and reused. As one example, QP and CBP data,MV reliability data, or EC reliability data that have been determinedfor a particular reference frame used for interpolation or extrapolationof one substitute frame may be reused if the same reference frame isused for another substitute frame.

Quality score calculator 175 may be configured to generate an overallquality score for a candidate reference frame based on the outputs ofone or more of objective metric checker 173, EC checker 181, QP checker181, and CBP checker 183. In a manner similar to the process describedwith reference to FIGS. 5 and 9, the individual scores produced byobjective metric checker 173, EC checker 181, QP checker 181, and CBPchecker 183 may be constructed simply as high or low quality values, orhigh, medium or low quality values, or computed along numerousgradations or a substantially continuous scale. Quality score calculator175 may accord equal or uneven weighting to each of the individualscores, e.g., such that the total quality score is a weighted summationof all of the scores.

The quality characteristics that are used to calculate the total score,e.g., objective metrics, EC characteristics, QP and CBP characteristics,may indicate whether the substitute frame is likely to produce anacceptable visual quality level when presented by display 170. Compareunit 177 may compare the total score to a quality threshold value. Ifthe total score is satisfactory, e.g., meets or exceeds the qualitythreshold value, compare unit 177 indicates that the substitute framehas a level of quality that is acceptable for presentation by display170. If the total score is less than the quality threshold value, thencompare unit 177 determines that the substitute frame is not acceptablefor presentation by display 170. In each case, analysis unit 168 maythen proceed to analyze quality for selective display of the nextsubstitute frame.

Even if the quality score indicates that quality is satisfactory, MVreliability checker 185 may be provided to determine whether motionvectors associated with one or more reference frames used to produce thesubstitute frame are reliable. MV reliability checker 185 may analyzethe reliability of motion vectors in one or more reference frames toensure that the selected reference frame(s) are likely to producequality frame substitution results if frame substitution makes use of amotion-compensated prediction method for interpolation or extrapolation.If the motion vectors are reliable, MV reliability checker 185 maytransmit an enable signal to control unit 170 to cause the substituteframe to be displayed by display 170. If the motion vectors are notreliable, however, MV reliability checker 185 may transmit a signal tocontrol unit 172 so that the substitute frame is not displayed bydisplay 170. Instead, control unit 172 may disable display of thesubstitute frame, such that display 170 may repeat a previous or futureframe in place of the substitute frame.

FIG. 15 is a flow diagram illustrating an example technique forgenerating quality scores to support selective display of substituteframes. The process of FIG. 15 may be similar to the process of FIG. 9.However, as described with reference to FIGS. 12-14, the scoring processmay be used for selective display of substitute frames that have alreadybeen interpolated or extrapolated, instead of for selection of referenceframes to perform interpolation or extrapolation. As shown in FIG. 15,in considering whether to display a substitute frame, analysis unit 168may retrieve one or more reference frames used for interpolation orextrapolation to produce the substitute frame (190) from received framebuffer 160, and analyze the QP and CBP values for the reference frame.As in the example of FIG. 9, as shown in FIG. 15, analysis unit 168 maygenerate a combined QP-based score based on the QP and CBP values. Ifthe QP value is less than an applicable QP threshold value (QP_th) andthe CBP value is not equal to zero (192), analysis unit 168 sets aQP-based score for the frame to “HIGH” (194). If either the QP value isgreater than or equal to the QP threshold value (QP_th) or the CBP valueis approximately equal to zero (192), analysis unit 42 sets the QP-basedscore for the frame to “LOW” (196).

As further shown in FIG. 15, analysis unit 168 also may be configured tocheck the reference frame or frames for slice loss (198). With referenceto FIG. 14B, slice loss checking may be performed by EC checker 179. Ifthere is a slice loss, analysis unit 168 may determine whether anadequate error concealment (EC) mechanism is available to correct forthe slice loss (200). If not, analysis unit 168 sets an EC-based scorefor the reference frame to “LOW” (202). If a reliable and adequate ECmechanism is available (124), analysis unit 168 sets the EC-based scoreto “HIGH” (204). In this case, if the lost slice can be reliablyreproduced using the EC mechanism, the quality of the reference framemay indicate that the substitute frame is reliable. If the slice cannotbe reproduced, however, the reference frame may indicate that thesubstitute frame could include erroneous information that should not bedisplayed.

Analysis unit 168 also may apply objective quality metrics to thereference frame(s) and/or the substitute frame (206). For example,analysis unit 168 may analyze the decoded and reconstructed version ofthe reference frame(s) to analyze any of a variety of objective metricssuch as blockiness, blurriness, color bleeding or the like. In addition,analysis unit 168 may analyze the pixel domain values of the substituteframe. Hence, the objective metrics may be applied to the referenceframe(s) and/or substitute frame in the pixel domain to evaluate visualquality. Analysis unit 168 may quantify each of the objective metrics toformulate a quality metric score QM. If the quality metric score QM isgreater than or equal to an applicable quality metric threshold (QM_th)(208), analysis unit 168 sets a QM-based score for the substitute frameto “HIGH” (210). If the quality metric score QM is less than or equal tothe quality metric threshold (QM_th), analysis unit 168 sets theQM-based score to “LOW” (212) for the substitute frame.

Analysis unit 168 may set a total quality score for the substitute frame(214) based on a combination of the QP-based score, the EC-based scoreand the QM-based score. The total quality score may be formulatedaccording to a weighted summation of the QP-based score, the EC-basedscore and the QM-based score for the reference frame(s) and/orsubstitute frames, where analysis unit 168 may attribute differentweights to each of the individual scores. The order in which theindividual QP-based score, the EC-based score and the QM-based score areobtained may be different from the order shown in FIG. 15. In addition,the particular types of scores shown in FIG. 15 may be subject tovariation. In general, analysis unit 168 may be configured to obtainquality scores that indicate a likelihood that a substitute frame willnot undermine visual quality and will contribute a degree of visualquality enhancement that justifies the expenditure of additionalresources to display the substitute frame.

FIG. 16 is a flow diagram illustrating an example technique for motionand/or quality analysis of reference video units to support referencevideo unit selection for video unit substitution when the video unitssupport a delay-sensitive video application. The flow diagram of FIG. 16corresponds substantially to the flow diagram of FIG. 8, but furtherincludes the operations of determining whether the video applicationsupported by decoder 14 is a delay sensitive application (216), such asvideo telephony and, if so, determining whether applicable temporaldistance criteria are met (218).

In the example of FIG. 16, when a candidate reference frame is retrieved(94), and a delay-sensitive application is detected (216), decoder 14may determine whether the frame is a future frame relative to the frameto be added. If so, decoder 14 determines whether the distance of thefuture reference frame is less than a threshold distance. If thecandidate reference frame is a previous frame or a future frame that isless than the threshold distance away from the frame to be added, thedistance criteria are met (218). In some aspects, the distance may beexpressed in terms of a number of frames separating the candidatereference frame and the frame to be added.

When the temporal distance criteria are met, decoder 14 may proceed withthe quality analysis, e.g., as described with reference to FIG. 8 andshown in operations (96)-(112). If the distance criteria are not met,however, decoder 14 may proceed to remove the candidate reference framefrom consideration as a reference frame for interpolation, and proceedto retrieve the next candidate reference frame (94) for consideration.Hence, the application of distance criteria for a delay-sensitiveapplication may be subject to a variety of different implementationssuch as control of selection unit 44 (FIG. 4) to exclude particularreference frames based on temporal distance from the substitute frame,calculation of quality scores based in part on distance via distanceunit 63 (FIG. 5), or control of analysis unit 42 to disable the qualityanalysis when the distance criteria are not met, e.g., as shown in FIG.16.

In general, techniques for selecting reference frames for interpolation,as described in this disclosure, may provide a FRUC implementation that,in various aspects, may lower resource consumption by disabling framesubstitution when not necessary or advantageous, and enhance the qualityof interpolated frames by selecting good reference frame for use inframe substitution. Advantages of enhanced quality may be achieved byanalyzing QP and CBP values, and selecting high quality reference framesbased on this analysis, which may be especially useful in videoscompressed with variable bit rate (VBR) control mechanisms.

In addition, enhanced quality may have achieved by not selecting frameswith slice or frames losses as reference frames if there is no errorconcealment mechanism, or if the error concealment mechanism does notprovide good quality frames. Avoidance of frames with slice or framelosses and inadequate error concealment may be especially useful whentransmission losses occur such as in the case of video telephonyapplications. A resource-focused mode also may be effective in reducingpower consumption, or consumption of other resources, whilesubstantially maintaining objective and subjective video quality inlow-motion video clips.

The techniques described herein may be implemented in hardware,software, firmware, or any combination thereof Any features described asmodules, units or components may be implemented together in anintegrated logic device or separately as discrete but interoperablelogic devices. In some cases, various features may be implemented as anintegrated circuit device, such as an integrated circuit chip orchipset. If implemented in software, the techniques may be realized atleast in part by a computer-readable medium comprising instructionsthat, when executed, cause a processor to perform one or more of themethods described above.

A computer-readable medium may form part of a computer program product,which may include packaging materials. A computer-readable medium maycomprise a computer data storage medium such as random access memory(RAM), synchronous dynamic random access memory (SDRAM), read-onlymemory (ROM), non-volatile random access memory (NVRAM), electricallyerasable programmable read-only memory (EEPROM), FLASH memory, magneticor optical data storage media, and the like. The techniquesadditionally, or alternatively, may be realized at least in part by acomputer-readable communication medium that carries or communicates codein the form of instructions or data structures and that can be accessed,read, and/or executed by a computer.

The code or instructions may be executed by one or more processors, suchas one or more DSPs, general purpose microprocessors, ASICs, fieldprogrammable logic arrays (FPGAs), or other equivalent integrated ordiscrete logic circuitry. Accordingly, the term “processor,” as usedherein may refer to any of the foregoing structure or any otherstructure suitable for implementation of the techniques describedherein. In addition, in some aspects, the functionality described hereinmay be provided within dedicated software modules or hardware modules.The disclosure also contemplates any of a variety of integrated circuitdevices that include circuitry to implement one or more of thetechniques described in this disclosure. Such circuitry may be providedin a single integrated circuit chip or in multiple, interoperableintegrated circuit chips in a so-called chipset. Such integrated circuitdevices may be used in a variety of applications, some of which mayinclude use in wireless communication devices, such as mobile telephonehandsets.

Various aspects of the disclosed techniques have been described. Theseand other aspects are within the scope of the following claims.

The invention claimed is:
 1. A method comprising: decoding, by a videodecoding device, a plurality of candidate reference video units;determining, by the video decoding device, to enable generation of anadditional video unit between two consecutive video units of theplurality of candidate reference video units, wherein the plurality ofcandidate reference video units includes at least one candidatereference video unit in addition to the two consecutive video units; inresponse to the determination to enable generation of the additionalvideo unit, analyzing, by the video decoding device, at least onecharacteristic specified in a respective syntax element of each of thecandidate reference video units, wherein the at least one characteristicincludes a characteristic indicating a quality of a corresponding one ofthe candidate reference video units and indicating a likely quality of agenerated video unit as generated from the corresponding one of thecandidate reference video units prior to initiation of a process togenerate the additional video unit from the corresponding one of thecandidate reference video units; and selecting, by the video decodingdevice, one or more of the candidate reference video units as areference video unit for interpolation or extrapolation of theadditional video unit based at least in part on the analysis, wherein atleast one of the selected candidate reference video units is differentthan the two consecutive video units between which the additional videounit is to be generated.
 2. The method of claim 1, wherein each of thecandidate reference video units is a candidate reference video frame andthe additional video unit is an additional video frame.
 3. The method ofclaim 1, wherein analyzing the at least one characteristic includesanalyzing a syntax element specifying a quantization parameter (QP) or asyntax element specifying coded block pattern (CBP) values of one ormore of the candidate reference video units.
 4. The method of claim 1,wherein analyzing the at least one characteristic further includesdetecting a video unit loss in one or more of the candidate referencevideo units, and wherein selecting one or more of the candidatereference video units as a reference video unit includes selecting oneor more of the candidate reference video units based on a quality oferror concealment.
 5. The method of claim 1, wherein analyzing the atleast one characteristic further includes analyzing one or moreobjective visual quality metrics of one or more of the candidatereference video units.
 6. The method of claim 1, wherein selecting oneor more of the candidate reference video units as a reference video unitcomprises selecting one or more candidate reference video units forwhich the quality level satisfies at least one threshold.
 7. The methodof claim 6, further comprising ranking, based on the quality level, atleast some of the candidate reference video units that satisfy the atleast one threshold, and selecting one or more of the candidatereference video units that are most highly ranked.
 8. The method ofclaim 6, further comprising adjusting the at least one threshold basedon a level of available power resources.
 9. The method of claim 1, themethod further comprising interpolating or extrapolating the additionalvideo unit based on the analysis.
 10. The method of claim 1, wherein theat least one characteristic further comprises a second characteristicindicating a temporal distance of one or more of the candidate referencevideo units from the additional video unit.
 11. The method of claim 10,further comprising: detecting a delay-sensitive video application; andselecting the one or more of the candidate reference video units basedat least in part on the temporal distance of one or more of thecandidate reference video units from the additional video unit when thedelay-sensitive video application is detected.
 12. The method of claim10, further comprising selecting the one or more of the candidatereference video units based at least in part on the temporal distance ofone or more of the candidate reference video units from the additionalvideo unit.
 13. The method of claim 1, wherein the at least onecharacteristic includes a quantization parameter (QP) and acharacteristic indicating a temporal distance of one or more of thecandidate reference video units from the additional video unit.
 14. Adevice comprising: a video decoder that decodes a plurality of candidatereference video units; an analysis unit that determines to enablegeneration of an additional video unit between two consecutive videounits of the plurality of candidate reference video units, wherein theplurality of candidate reference video units includes at least onecandidate reference video unit in addition to the two consecutive videounits, and, in response to the determination to enable generation of theadditional video unit, analyzes at least one characteristic specified ina respective syntax element of each of the candidate reference videounits, wherein the at least one characteristic includes a characteristicindicating a quality of a corresponding one of the candidate referencevideo units and indicating a likely quality of a generated video unit asgenerated from the corresponding one of the candidate reference videounits prior to initiation of a process to generate the additional videounit from the corresponding one of the candidate reference video units;and a selection unit that selects one or more of the candidate referencevideo units as a reference video unit for interpolation or extrapolationof the additional video unit based at least in part on the analysis,wherein at least one of the selected candidate reference video units isdifferent than the two consecutive video units between which theadditional video unit is to be generated.
 15. The device of claim 14,wherein each of the candidate reference video units is a candidatereference video frame and the additional video unit is an additionalvideo frame.
 16. The device of claim 14, wherein the analysis unitanalyzes at least one of a syntax element specifying a quantizationparameter (QP) or a syntax element specifying coded block pattern (CBP)values of one or more of the candidate reference video units.
 17. Thedevice of claim 14, wherein the analysis unit detects video unit loss inone or more of the candidate reference video units, and the selectionunit selects one or more of the candidate reference video units based ona quality of error concealment.
 18. The device of claim 14, wherein theanalysis unit analyzes one or more objective visual quality metrics ofone or more of the candidate reference video units.
 19. The device ofclaim 14, wherein the selection unit selects one or more candidatereference video units for which the quality level satisfies at least onethreshold.
 20. The device of claim 19, wherein the analysis unit ranks,based on the quality level, at least some of the candidate referencevideo units that satisfy the at least one threshold, and the selectionunit selects one or more of the candidate reference video units that aremost highly ranked.
 21. The device of claim 19, further comprising anadjustment unit that adjusts the at least one threshold based on a levelof available power resources.
 22. The device of claim 14, the devicefurther comprising a substitution unit that interpolates or extrapolatesthe additional video unit based on the analysis.
 23. The device of claim14, wherein the at least one characteristic further comprises a secondcharacteristic indicating a temporal distance of one or more of thecandidate reference video units from the additional video unit.
 24. Thedevice of claim 23, further comprising a delay detection unit thatdetects a delay-sensitive video application, wherein the selection unitselects the one or more of the candidate reference video units based atleast in part on the temporal distance of the one or more candidatereference video units from the additional video unit when thedelay-sensitive video application is detected.
 25. The device of claim23, wherein the selection unit selects the one or more of the candidatereference video units based at least in part on the temporal distance ofthe one or more candidate reference video units from the additionalvideo unit.
 26. The device of claim 14, wherein the device comprises awireless communication device handset.
 27. The device of claim 14,wherein the device comprises an integrated circuit device.
 28. Thedevice of claim 14, wherein the at least one characteristic includes aquantization parameter (QP) and a characteristic indicating a temporaldistance of one or more of the candidate reference video units from theadditional video unit.
 29. A device comprising: means for decoding aplurality of candidate reference video units; means for determining toenable generation of an additional video unit between two consecutivevideo units of the plurality of candidate reference video units, whereinthe plurality of candidate reference video units includes at least onecandidate reference video unit in addition to the two consecutive videounits; means for analyzing, in response to the determination to enablegeneration of the additional video unit, at least one characteristicspecified in a respective syntax element of each of the candidatereference video units, wherein the at least one characteristic includesa characteristic indicating a quality of a corresponding one of thecandidate reference video units and indicating a likely quality of agenerated video unit as generated from the corresponding one of thecandidate reference video units prior to initiation of a process togenerate the additional video unit from the corresponding one of thecandidate reference video units; and means for selecting one or more ofthe candidate reference video units as a reference video unit forinterpolation or extrapolation of the additional video unit based atleast in part on the analysis, wherein at least one of the selectedcandidate reference video units is different than the two consecutivevideo units between which the additional video unit is to be generated.30. The device of claim 29, wherein each of the candidate referencevideo units is a candidate reference video frame and the additionalvideo unit is an additional video frame.
 31. The device of claim 29,wherein the means for analyzing the at least one characteristic includesmeans for analyzing at least one of a syntax element specifying aquantization parameter (QP) or a syntax element specifying coded blockpattern (CBP) values of one or more of the candidate reference videounits.
 32. The device of claim 29, wherein the means for analyzing theat least one characteristic further includes means for detecting videounit loss in one or more of the candidate reference video units, and themeans for selecting includes means for selecting one or more of thecandidate reference video units based on a quality of error concealment.33. The device of claim 29, wherein the means for analyzing the at leastone characteristic further includes means for analyzing one or moreobjective visual quality metrics of one or more of the candidatereference video units.
 34. The device of claim 29, wherein the means forselecting comprises means for selecting one or more candidate referencevideo units for which the quality level satisfies at least onethreshold.
 35. The device of claim 34, further comprising means forranking, based on the quality level, at least some of the candidatereference video units that satisfy the at least one threshold, and meansfor selecting one or more of the candidate reference video units thatare most highly ranked.
 36. The device of claim 34, further comprisingmeans for adjusting the at least one threshold based on a level ofavailable power resources.
 37. The device of claim 29, the devicefurther comprising means for interpolating or extrapolating theadditional video unit based on the analysis.
 38. The device of claim 29,wherein the at least one characteristic further comprises a secondcharacteristic indicating a temporal distance of one or more of thecandidate reference video units from the additional video unit.
 39. Thedevice of claim 38, further comprising: means for detecting adelay-sensitive video application; and means for selecting the one ormore of the candidate reference video units based at least in part onthe temporal distance of the one or more candidate reference video unitsfrom the additional video unit when the delay-sensitive videoapplication is detected.
 40. The device of claim 38, further comprisingmeans for selecting the one or more of the candidate reference videounits based at least in part on the temporal distance of the one or morecandidate reference video units from the additional video unit.
 41. Thedevice of claim 29, wherein the at least one characteristic includes aquantization parameter (QP) and a characteristic indicating a temporaldistance of one or more of the candidate reference video units from theadditional video unit.
 42. A non-transitory computer-readable mediumcomprising instructions to cause one or more processors to: decode aplurality of candidate reference video units; determine to enablegeneration of an additional video unit between two consecutive videounits of the plurality of candidate reference video units, wherein theplurality of candidate reference video units includes at least onecandidate reference video unit in addition to the two consecutive videounits; analyze, in response to the determination to enable generation ofthe additional video unit, at least one characteristic specified in arespective syntax element of each of the candidate reference videounits, wherein the at least one characteristic includes a characteristicindicating a quality of a corresponding one of the candidate referencevideo units and indicating a likely quality of a generated video unit asgenerated from the corresponding one of the candidate reference videounits prior to initiation of a process to generate the additional videounit from the corresponding one of the candidate reference video units;and select one or more of the candidate reference video units as areference video unit for interpolation or extrapolation of theadditional video unit based at least in part on the analysis, wherein atleast one of the selected candidate reference video units is differentthan the two consecutive video units between which the additional videounit is to be generated.
 43. The non-transitory computer-readable mediumof claim 42, wherein each of the candidate reference video units is acandidate reference video frame and the additional video unit is anadditional video frame.
 44. The non-transitory computer-readable mediumof claim 42, wherein the instructions cause one or more processors toanalyze at least one of a syntax element specifying a quantizationparameter (QP) or a syntax element specifying coded block pattern (CBP)values of one or more of the candidate reference video units.
 45. Thenon-transitory computer-readable medium of claim 42, wherein theinstructions cause one or more processors to detect video unit loss inone or more of the candidate reference video units, and select one ormore of the candidate reference video units based on a quality of errorconcealment.
 46. The non-transitory computer-readable medium of claim42, wherein the instructions cause one or more processors to analyze oneor more objective visual quality metrics of one or more of the candidatereference video units.
 47. The non-transitory computer-readable mediumof claim 42, wherein the instructions cause the one or more processorsto select one or more candidate reference video units for which thequality level satisfies at least one threshold.
 48. The non-transitorycomputer-readable medium of claim 47, wherein the instructions cause theone or more processors to select rank, based on the quality level, atleast some of the candidate reference video units that satisfy the atleast one threshold, and select one or more of the candidate referencevideo units that are most highly ranked.
 49. The non-transitorycomputer-readable medium of claim 47, wherein the instructions cause theone or more processors to adjust the at least one threshold based on alevel of available power resources.
 50. The non-transitorycomputer-readable medium of claim 42, wherein the instructions cause theone or more processors to interpolate or extrapolate the additionalvideo unit based on the analysis.
 51. The non-transitorycomputer-readable medium of claim 42, wherein the at least onecharacteristic further comprises a second characteristic indicating atemporal distance of one or more of the candidate reference video unitsfrom the additional video unit.
 52. The non-transitory computer-readablemedium of claim 51, wherein the instructions cause the one or moreprocessors to: detect a delay-sensitive video application; and selectthe one or more of the candidate reference video units based at least inpart on the temporal distance of one or more of the candidate referencevideo units from the additional video unit when the delay-sensitivevideo application is detected.
 53. The non-transitory computer-readablemedium of claim 51, wherein the instructions cause the one or moreprocessors to select the one or more of the candidate reference videounits based at least in part on the temporal distance of one or more ofthe candidate reference video units from the additional video unit. 54.The non-transitory computer-readable storage medium of claim 42, whereinthe at least one characteristic includes a quantization parameter (QP)and a characteristic indicating a temporal distance of one or more ofthe candidate reference video units from the additional video unit.